Want to give up work and live off passive income? I do. That’s why I’m trying to build an investment pot large enough to buy lots of dividend shares that will help me retire early.

But life is expensive. No matter what age you are there are all sorts of living costs that need to be met. With this in mind, how much would you need in a Stocks and Shares ISA to try and replace your employment earnings with an identical income stream from dividends? Let’s see.

A crucial difference

According to government figures, the average (median) UK salary is £32,890. So that’s our passive income target.

But we are not comparing like with like here. Anyone with an ISA will be able to earn dividends tax-free. By contrast, salaries over £12,570 are usually taxed.

Ignoring this important distinction, a portfolio of dividend shares paying 5% a year would need to worth £657,800 to match the average salary. At 6%, this falls to £548,167. With an annual return of 7%, a £469,857 ISA would suffice.

Please note that tax treatment depends on the individual circumstances of each client and may be subject to change in future. The content in this article is provided for information purposes only. It is not intended to be, neither does it constitute, any form of tax advice. Readers are responsible for carrying out their own due diligence and for obtaining professional advice before making any investment decisions.

Don’t panic!

These are significant sums.

But I reckon it’s possible to achieve something similar by investing little and often in growth shares. Generally speaking, these are likely to beat the returns of the wider market as their businesses are expanding more rapidly.

They could be, for example, in the tech sector or life sciences. Or they might be a successful investment company or a business that’s growing quickly through acquisition. Alternatively, they could simply be operating in the right sector at the right time.  

The table below identifies five FTSE 100 stocks that have experienced rapid share price growth since April 2021. Their five-year average annual return has been an impressive 15.3%.

Of course, there are no guarantees that history will be repeated. And I would have to do more research before deciding whether they are worth considering for the next five years. Indeed, this is a random list based on some of the examples I gave earlier. There are plenty of others I could have chosen.

But the purpose of the exercise is to illustrate the potential returns available from investing in the UK stock market.

For example, someone putting £150 a month into an ISA for 25 years, would see it grow to approximately £514,000, assuming an annual growth rate of 15.3%.

Something else to consider

The boss of Standard Life (LSE:SDLF), the pension and savings group, told the BBC this week (16 April) that only one in seven of us are saving enough for a decent retirement.

Ironically, with a yield of 7.4%, I think those wanting to improve their living standard in old age could consider the group’s shares. Applying this rate to our £514,000 ISA would produce dividends of £38,036, more than the country’s average salary.

However, dividends cannot be guaranteed. Threats to its payout include increased competition as well as global economic uncertainty. The group’s £309bn investment portfolio is likely to suffer heavy losses during periods of market instability.   

But 2025 was a good year. Adjusted operating profit increased 15% compared to 2024. And its Solvency II ratio (a measure of balance sheet strength) went up. This week’s announcement that it’s agreed to buy Aegon’s UK pension business should help underpin future increases in its earnings and dividend.