After a fair amount of work, I was able to get it to build on Ubuntu 12.04 x86 and Debian 7.4 x86_64. I wrote up a guide below. Can you please try following it to see if it resolves the issue?

If not please let me know where you get stuck.

Install Common Dependencies

sudo apt-get install build-essential autoconf libtool pkg-config python-opengl python-imaging python-pyrex python-pyside.qtopengl idle-python2.7 qt4-dev-tools qt4-designer libqtgui4 libqtcore4 libqt4-xml libqt4-test libqt4-script libqt4-network libqt4-dbus python-qt4 python-qt4-gl libgle3 python-dev

Install NumArray 1.5.2

wget https://tinyurl.com/ywzmahjf -O numarray-1.5.2.tgz
tar xfvz numarray-1.5.2.tgz
cd numarray-1.5.2
sudo python setup.py install

Install Numeric 23.8

wget http://downloads.sourceforge.net/project/numpy/OldFiles/23.8/Numeric-23.8.tar.gz?r&ts=1395946715&use_mirror=iweb -O numeric-23.8.tgz
tar xfvz numeric-23.8.tgz
cd Numeric-23.8
sudo python setup.py install

Install HDF5 1.6.5

wget ftp://ftp.hdfgroup.org/HDF5/releases/hdf5-1.6/hdf5-1.6.5.tar.gz
tar xfvz hdf5-1.6.5.tar.gz
cd hdf5-1.6.5
./configure --prefix=/usr/local
sudo make 
sudo make install

Install Nanoengineer

git clone https://github.com/kanzure/nanoengineer.git
cd nanoengineer
./bootstrap
./configure
make
sudo make install

Troubleshooting

On Debian Jessie, you will receive the error message that cant pants mentioned. There seems to be an issue in the automake scripts. x86_64-linux-gnu-gcc is inserted in CFLAGS and gcc will interpret that as a name of one of the source files. As a workaround, let's create an empty file with that name. Empty so that it won't change the program and that very name so that compiler picks it up. From the cloned nanoengineer directory, run this command to make gcc happy (it is a hack yes, but it does work) ...

touch sim/src/x86_64-linux-gnu-gcc

If you receive an error message when attemping to compile HDF5 along the lines of: "error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT in second argument needs 3 arguments", then modify the file perform/zip_perf.c, line 548 to look like the following and then rerun make...

output = open(filename, O_RDWR | O_CREAT, S_IRUSR|S_IWUSR);

If you receive an error message about Numeric/arrayobject.h not being found when building Nanoengineer, try running

export CPPFLAGS=-I/usr/local/include/python2.7
./configure
make
sudo make install

If you receive an error message similar to "TRACE_PREFIX undeclared", modify the file sim/src/simhelp.c lines 38 to 41 to look like this and re-run make:

#ifdef DISTUTILS
static char tracePrefix[] = "";
#else
static char tracePrefix[] = "";

If you receive an error message when trying to launch NanoEngineer-1 that mentions something similar to "cannot import name GL_ARRAY_BUFFER_ARB", modify the lines in the following files

/usr/local/bin/NanoEngineer1_0.9.2.app/program/graphics/drawing/setup_draw.py
/usr/local/bin/NanoEngineer1_0.9.2.app/program/graphics/drawing/GLPrimitiveBuffer.py
/usr/local/bin/NanoEngineer1_0.9.2.app/program/prototype/test_drawing.py

that look like this:

from OpenGL.GL import GL_ARRAY_BUFFER_ARB
from OpenGL.GL import GL_ELEMENT_ARRAY_BUFFER_ARB

to look like this:

from OpenGL.GL.ARB.vertex_buffer_object import GL_ARRAY_BUFFER_AR
from OpenGL.GL.ARB.vertex_buffer_object import GL_ELEMENT_ARRAY_BUFFER_ARB

I also found an additional troubleshooting text file that has been removed, but you can find it here

I encountered the same problem in college having installed Linux Mint for the main project of my final year, the third solution below worked for me.

When encountering this error please note before the error it may say you are missing a package or header file — you should find those and install them and verify if it works (e.g. ssl → libssl).

For Python 2.x use:

sudo apt-get install python-dev

For Python 2.7 use:

sudo apt-get install libffi-dev

For Python 3.x use:

sudo apt-get install python3-dev

or for a specific version of Python 3, replace x with the minor version in

sudo apt-get install python3.x-dev

# ls /usr/bin/x86* -l
lrwxrwxrwx 1 root root    7 Jun  3 01:27 /usr/bin/x86_64 -> setarch
lrwxrwxrwx 1 root root    7 Jun  3 01:27 /usr/bin/x86_64-linux-gnu-cpp -> cpp-4.7
lrwxrwxrwx 1 root root    7 Jan 30 13:55 /usr/bin/x86_64-linux-gnu-cpp-4.4 -> cpp-4.4
lrwxrwxrwx 1 root root    7 Apr 13 06:08 /usr/bin/x86_64-linux-gnu-cpp-4.6 -> cpp-4.6
lrwxrwxrwx 1 root root    7 Jun  3 01:27 /usr/bin/x86_64-linux-gnu-cpp-4.7 -> cpp-4.7
lrwxrwxrwx 1 root root    7 Apr 22 02:44 /usr/bin/x86_64-linux-gnu-g++ -> g++-4.7
lrwxrwxrwx 1 root root    7 Apr 15 07:12 /usr/bin/x86_64-linux-gnu-g++-4.7 -> g++-4.7
lrwxrwxrwx 1 root root    7 Apr 22 02:44 /usr/bin/x86_64-linux-gnu-gcc -> gcc-4.7
lrwxrwxrwx 1 root root    7 Jan 30 13:57 /usr/bin/x86_64-linux-gnu-gcc-4.4 -> gcc-4.4
lrwxrwxrwx 1 root root    7 Apr 13 06:10 /usr/bin/x86_64-linux-gnu-gcc-4.6 -> gcc-4.6
lrwxrwxrwx 1 root root    7 Jun  3 01:27 /usr/bin/x86_64-linux-gnu-gcc-4.7 -> gcc-4.7

... There is no difference it would seem; try changing the link and see if it works.

gcc-12 is not available in ubuntu 20.04, so we need to compile it from source code, here are the steps which I borrowed from this video:

• Step 1: clone gcc source code and checkout gcc-12 branch

$ git clone https://gcc.gnu.org/git/gcc.git gcc-source
$ cd gcc-source/
$ git branch -a
$ git checkout remotes/origin/releases/gcc-12

• Step 2: make another build dir

Note this is important as running ./configure from within the source directory is not supported as documented here.

$ mkdir ../gcc-12-build
$ cd ../gcc-12-build/
$ ./../gcc-source/configure --prefix=$HOME/install/gcc-12 --enable-languages=c,c++

• Step 3: installing GCC prequisites and run configure again

The missing libraries will be shown in above ./confgiure output, search and install them one by one.

$ apt-cache search MPFR
$ sudo apt-get install libmpfrc++-dev
$ apt-cache search MPC | grep dev
$ sudo apt-get install libmpc-dev
$ apt-cache search GMP | grep dev
$ sudo apt-get install libgmp-dev
$ sudo apt-get install gcc-multilib
$ ./../gcc-source/configure --prefix=$HOME/install/gcc-12 --enable-languages=c,c++

An alternative is to run the download_prerequisites script.

 cd gcc-source/
$ ./contrib/download_prerequisites
$ ./../gcc-source/configure --prefix=$HOME/install/gcc-12 --enable-languages=c,c++

• Step 4: compile gcc-12

$ make -j16

Still flex is missing:

$ sudo apt-get install flex
$ ./../gcc-source/configure --prefix=$HOME/install/gcc-12 --enable-languages=c,c++
$ make -j16
$ make install

Another way is to use Ubuntu 22.04 where gcc-12 is available. In Ubuntu 22.04, gcc-12 can be installed with apt:

$ sudo apt install gcc-12

NOTE: The 6.5 Kernel reports being built as x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) and this should be OK. This is because the kernel is not built on your system during install. See https://askubuntu.com/a/1171939/231142 for more info. During any builds it might give you a warning about not matching the installed version since they are not named the same. Installed version reports as gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04)

gcc-12 should already be installed, but if it isn't, you can install it or reinstall it by running:

sudo apt install --reinstall gcc-12

Then all you should have to do is update the link for gcc to go to the gcc-12 binary:

sudo ln -s -f /usr/bin/gcc-12 /usr/bin/gcc

You can set it back to 11 as well by doing the following:

sudo ln -s -f /usr/bin/gcc-11 /usr/bin/gcc

You can check it by running gcc --version.

terrance@terrance-ubuntu:~$ gcc --version
gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Try sudo apt-get install python3.X-dev where X is your python3 subversion.