The latest version of SQL Server (2012) permits the following.

SELECT 
    RowID, 
    Col1,
    SUM(Col1) OVER(ORDER BY RowId ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS Col2
FROM tablehh
ORDER BY RowId

or

SELECT 
    GroupID, 
    RowID, 
    Col1,
    SUM(Col1) OVER(PARTITION BY GroupID ORDER BY RowId ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS Col2
FROM tablehh
ORDER BY RowId

This is even faster. Partitioned version completes in 34 seconds over 5 million rows for me.

Thanks to Peso, who commented on the SQL Team thread referred to in another answer.

You need a "window" aggregate, i.e. an OVER clause in the aggregate. And because the query already has a GROUP BY, the aggregate needed is the SUM() over the COUNT(d.FK_User) you already have:

SELECT 
    u.Username, 
    -- COUNT(d.FK_User) AS UserCount,     -- if you need both counts
    SUM(COUNT(d.FK_User)) OVER (PARTITION BY u.Username
                                ORDER BY CAST(d.CreationTimestamp AS DATE)
                                ROWS BETWEEN UNBOUNDED PRECEDING
                                         AND CURRENT ROW) 
        AS CumulativeUserCount, 
    CAST(d.CreationTimestamp AS DATE) AS CreationDate
FROM data d 
    INNER JOIN users u 
    ON u.PK_User = d.FK_User
GROUP BY 
    CAST(d.CreationTimestamp AS DATE), u.Username
ORDER BY
    CreationDate ;

• The ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW is not strictly needed. The default frame for window aggregates when there is an ORDER BY inside OVER () is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which would produce the same results as ROWS but is likely to generate a suboptimal execution plan. See Aaron Bertrand's blog article Best approaches for running totals – updated for SQL Server 2012 for details.
• I've simplified the complicated expression that seems to just strip the time part form the datetime column.
• Removed the single quotes around the aliases. Aliases and identifiers better not be reserved keywords and not have special characters or spaces. If you do need any of these, then it's preferred to quote them with double quotes "Count" or square brackets [Count].

You can do this using a Tally Table:

SQL Fiddle

DECLARE @startDate  DATE,
        @endDate    DATE

SELECT
    @startDate  = MIN(deliver_date),
    @endDate    = MAX(deliver_date)
FROM TestTable

;WITH E1(N) AS(
    SELECT 1 FROM(VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1))t(N)
),
E2(N) AS(SELECT 1 FROM E1 a CROSS JOIN E1 b),
E4(N) AS(SELECT 1 FROM E2 a CROSS JOIN E2 b),
Tally(N) AS(
    SELECT TOP(DATEDIFF(DAY, @startDate, @endDate) + 1)
        ROW_NUMBER() OVER(ORDER BY (SELECT NULL))
    FROM E4
),
CteAllDates AS(
    SELECT
        deliver_date = DATEADD(DAY, t.N-1, @startDate),
        quantity = ISNULL(tt.quantity, 0)
    FROM Tally t
    LEFT JOIN TestTable tt
        ON DATEADD(DAY, N-1, @startDate) = tt.deliver_date
)
SELECT
    deliver_date,
    cumQty = SUM(quantity) OVER(ORDER BY deliver_date)
FROM CteAllDates

First, you want to generate all dates starting from the MIN(deliver_date) up to MAX(deliver_date). This is done using a tally table, the CTEs from E1(N) up to Tally(N).

Now that you have all the dates, do a LEFT JOIN on the original table, TestTable, to get the corresponding quantity, assigning 0 if there is no matching dates.

Lastly, to get the cumulative sum, you can use SUM(quantity) OVER(ORDER BY deliver_date).

For more explanation on tally table, see my answer here.

Try this: SUM(t1.NewSubs) over(order by t1.Date)

For example:

;WITH TBL AS
  (SELECT CONVERT(date,TakenOn) AS Date,
          COUNT(*) AS 'NewSubs'
   FROM Urls (NOLOCK)
   WHERE FId = 11
     AND (TakenOn >= @StartDate)
     AND (TakenOn <= @EndDate)
   GROUP BY CONVERT(date,TakenOn))
SELECT t1.Date, t1.NewSubs,
                SUM(t1.NewSubs) over(order by t1.Date) AS 'TotalSubs'
FROM TBL t1
INNER JOIN TBL t2 ON t1.Date >= t2.Date
GROUP BY t1.Date, t1.NewSubs

You have the wrong grouping clause in the reg CTE, you need cast([created_at] as date).

The outer group by is not necessary

The default window in an OVER clause (when there is an ORDER BY) is unfortunately RANGE UNBOUNDED PRECEDING, which is rarely what people expect.
You must specify ROWS UNBOUNDED PRECEDING explicitly.

with reg as(      
    select
        cast([created_at] as date) as [date],
        sum(cast([is_registered] as int)) as [sum_of_reg]
    FROM [dbo].[Registrations]
    group by cast([created_at] as date)
)
select
    [date],
    sum_of_reg,
    sum(sum_of_reg) over (order by [date] ROWS UNBOUNDED PRECEDING) as [running total]
FROM reg
order by [date] asc

You can even do this in a single level, by using a window function on top of normal aggregation.

select
    cast([created_at] as date) as [date],
    sum(cast([is_registered] as int)) as [sum_of_reg],
    sum(sum(cast([is_registered] as int)))
      over (order by [date] ROWS UNBOUNDED PRECEDING) as [running total]
FROM [dbo].[Registrations]
group by cast([created_at] as date)
order by [date] asc