Disclaimer 1: I'm not sure if your question is about how to calculate the counts per subgroup, or how to plot a 5-set Venn diagram. I'm assuming the latter.

Disclaimer 2: I find 5-set Venn diagrams extremely difficult to read. To the point of being useless. But that's my personal opinion.

If other R packages are an option, here is a worked-out 5-set example using VennDiagram (straight from the VennDiagram reference manual)

library(VennDiagram);
venn.plot <- draw.quintuple.venn(
    area1 = 301, area2 = 321, area3 = 311, area4 = 321, area5 = 301,
    n12 = 188, n13 = 191, n14 = 184, n15 = 177,
    n23 = 194, n24 = 197, n25 = 190,
    n34 = 190, n35 = 173, n45 = 186,
    n123 = 112, n124 = 108, n125 = 108,
    n134 = 111, n135 = 104, n145 = 104,
    n234 = 111, n235 = 107, n245 = 110,
    n345 = 100,
    n1234 = 61, n1235 = 60, n1245 = 59,
    n1345 = 58, n2345 = 57,
    n12345 = 31,
    category = c("A", "B", "C", "D", "E"),
    fill = c("dodgerblue", "goldenrod1", "darkorange1", "seagreen3", "orchid3"),
    cat.col = c("dodgerblue", "goldenrod1", "darkorange1", "seagreen3", "orchid3"),
    cat.cex = 2,
    margin = 0.05,
    cex = c(
        1.5, 1.5, 1.5, 1.5, 1.5, 1, 0.8, 1, 0.8, 1, 0.8, 1, 0.8, 1, 0.8,
        1, 0.55, 1, 0.55, 1, 0.55, 1, 0.55, 1, 0.55, 1, 1, 1, 1, 1, 1.5),
    ind = TRUE);

png("venn_5set.png");
grid.draw(venn.plot);
dev.off();

Update [15 November 2017]

Your source table is in an atypical format. As I explain in my comments, you usually start with either a binary matrix (one column per set, membership of every observation indicated by 0's or 1's), or a list of set elements.

To be honest, I'm less and less sure about what you are actually trying to do. I have a feeling that there might be a misconception about Venn diagrams. For example, let's take a look at the first rows of your table

# Read data
library(readxl);
data <- as.data.frame(read_excel("~/Downloads/dataset4venn.xlsx"));
rownames(data) <- data[, 1];
data <- data[, -1];
head(data);
#  A B  C D  E
#1 8 8  7 8 10
#2 0 0 10 0  2
#3 0 0  0 0  3
#4 0 0  1 2  0
#5 1 0  1 0  2
#6 0 0  0 0  1    

An observation is either the presence (encoded by 1) or the absence (encoded by 0) of a unique element (in your case a species) in a specific group (i.e. a sampling site). The number of sightings as you call it does not matter here: a Venn diagram explores the logical relations between different species sampled at different sites, or in other words which unique species are shared by sites A-E.

Having said that and ignoring the number of sightings per site, you can show overlaps between different sites in the following 5-set Venn diagram. I first define a helper function cts to calculate counts per group/overlap, and then feed those numbers into draw.quintuple.venn.

# Function to calculate the count per group/overlap
# Note: data is a global variable
cts <- function(set) {
    df <- data;
    for (i in 1:length(set)) df <- subset(df, df[set[i]] >= 1);
    nrow(df);
}

# Plot
library(VennDiagram);
venn.plot <- draw.quintuple.venn(
    area1 = cts("A"), area2 = cts("B"), area3 = cts("C"),
    area4 = cts("D"), area5 = cts("E"),
    n12 = cts(c("A", "B")), n13 = cts(c("A", "C")), n14 = cts(c("A", "D")),
    n15 = cts(c("A", "E")), n23 = cts(c("B", "C")), n24 = cts(c("B", "D")),
    n25 = cts(c("B", "E")), n34 = cts(c("C", "D")), n35 = cts(c("C", "E")),
    n45 = cts(c("D", "E")),
    n123 = cts(c("A", "B", "C")), n124 = cts(c("A", "B", "D")),
    n125 = cts(c("A", "B", "E")), n134 = cts(c("A", "C", "D")),
    n135 = cts(c("A", "C", "E")), n145 = cts(c("A", "D", "E")),
    n234 = cts(c("B", "C", "D")), n235 = cts(c("B", "C", "E")),
    n245 = cts(c("B", "D", "E")), n345 = cts(c("C", "D", "E")),
    n1234 = cts(c("A", "B", "C", "D")), n1235 = cts(c("A", "B", "C", "E")),
    n1245 = cts(c("A", "B", "D", "E")), n1345 = cts(c("A", "C", "D", "E")),
    n2345 = cts(c("B", "C", "D", "E")),
    n12345 = cts(c("A", "B", "C", "D", "E")),
    category = c("A", "B", "C", "D", "E"),
    fill = c("dodgerblue", "goldenrod1", "darkorange1", "seagreen3", "orchid3"),
    cat.col = c("dodgerblue", "goldenrod1", "darkorange1", "seagreen3", "orchid3"),
    cat.cex = 2,
    margin = 0.05,
    cex = c(
        1.5, 1.5, 1.5, 1.5, 1.5, 1, 0.8, 1, 0.8, 1, 0.8, 1, 0.8, 1, 0.8,
        1, 0.55, 1, 0.55, 1, 0.55, 1, 0.55, 1, 0.55, 1, 1, 1, 1, 1, 1.5),
    ind = TRUE);

png("venn_5set.png");
grid.draw(venn.plot);
dev.off();

PS

Various R packages/internet sources offer helper functions to calculate overlaps based on e.g. a binary matrix or a list of set elements. For example, the R/Bioconductor package limma offers a function limma::vennCounts that calculates counts for all overlaps based on a binary matrix. So if you don't want to write your own function (like I did), you can also use those. Either way, in the case of more complex Venn diagrams, I would suggest to not calculate overlaps manually by hand, as it's easy to make a mistake (see your error message).