One thing you may want to look at. I remember encountering a bug in one of the Borland products in that they complained of being out of memory on a 640K, himem-aware, DOS 6.22 system.
That was due to the fact it used a signed comparison sequence rather than unsigned (e.g., jl rather than jb ). This led to the problem that anything greater than 512K available indicated a negative value. So, if the program needed 128K and erroneously calculated it had -32K free, it would assume not enough memory.
To fix this, I had to track down the jump instruction and change it to the correct type.
I have no idea if this is the same cause of the problem that you're seeing but, given GW-BASIC was first introduced with MSDOS 1 (I think) and was supplanted by QBasic in MSDOS 5, it's possible it never ran in a system where more than 512k was free. Your comment that it ran on a system "that had only 512KB of conventional memory" also seems to gel with this possibility.
One way to check if this is the cause is to reduce your memory supply to below the 512K barrier and see if it magically starts working.
Finding the errant instruction (in the Borland product) wasn't easy but was eventually achieved by meticulously disassembling and testing changes and, eventually, using something like the Sourcer intelligent disassembler which made it much easier (assuming that you can still find it).
The task may be less work in this case since at least one version of the program is available on GitHub . However, there's probably a big difference between that version and 3.10.
In fact, initial analysis of that early version seems to indicate the 512K boundary is unlikely to be the problem - the "Out of memory" message only shows up during FDB allocation: INIFDB calls FALLOC which generates that message if there's not enough room between the pointer to the free string space and the end of that string space.
It tries once, then retries after garbage collection if first attempt failed, then generates that error if it still fails.
That limited error generation scenario seems strange but there may be other occurrences "hidden" in the fact this code was automatically translated from an earlier (8080?) version so includes such beauties as INS86 71, 36, FRETOP instead of real 8086 mnemonics like CMP FRETOP, BX .
In any case, this (translated) version very much uses segments so it's likely that you'll only have a single segment for code/data/stack. In other words, its limits will be related to 64K rather than 640K.
You should also take note of the information on the GW-BASIC Wikipedia page :
Microsoft did not offer a generic version of MS-DOS until v3.20 in 1986; before then, all variants of the operating system were OEM versions.
Some variants of BASIC has extra features to support a particular machine. For example, the AT&T and Tandy versions of DOS include a special GW-BASIC that supports their enhanced sound and graphics capabilities.
On the assumption that GW-BASIC versions tracked the MSDOS version number, that means your version 3.10 is an OEM-specific version so may well have very hardware-specific stuff embedded in it. You would probably be better moving to at least the generic 3.20 (or the latest available, 3.23).
That might also explain why it works in DOSBox and not on raw hardware. Since the former is meant to provide the ability to run a great many older programs, it may well have far more (emulated) hardware options than a normal system.
On your comment that GW-BASIC ran fine on DOS 3.3, that may well be true. It's doubtful Microsoft expended any effort in getting GW-BASIC running under MSDOS 5+ since QBasic was by then the anointed method.
Hence, any bug report where it may not run under MSDOS 6 would probably have been rejected outright by Microsoft. If it runs in the same MSDOS version under DOSBox, that may be entirely by accident :-)
In fact, I'm not even sure it makes sense to think about what version of MSDOS is running in DOSBox. I seem to recall it doesn't actually run a real version, instead opting for running its own code that faithfully reproduces the behaviour. From the Wikipedia page for DOSBox (my emphasis):
DOSBox is a full-system emulator that provides BIOS interrupts and contains its own internal DOS-like shell. This means that it can be used without owning a license to any real DOS operating system.
I suppose the bottom line there is that it's not that the memory requirements are an issue, but that the ability to run it at all (at least in a supported manner) is the problem.
