includer.py

Copyright Michael Foord
You are free to modify, use and relicense this code.
No warranty express or implied for the accuracy, fitness to purpose or otherwise for this code....
Use at your own risk !!!

Maintained at www.voidspace.org.uk/atlantibots/pythonutils.html

Implements a recursive include command - with nifty features like an additional incdir command, will replace 'from module import... commands' and will remove stuff below 'if __name__ ==...' lines in included modules.

python includer.py infilename outfilename
## INCLUDE filename
## INCDIR 'path name'


This program allows you to include scripts into other scripts.
It is written to make it easier to distribute scripts as single files -  but maintain as seperate modules.
As an example, my ConfigObj module depends on the listquote and caseless modules for some of it's core functionality.
Both those modules are useful in their own right and are maintained seperately.
includer allows me to distribute ConfigObj as a single file (fullconfigobj.py) and I can painlessly create a 'current version' anytime by running includer on the main configobj file - which creates a new file that contains the latest version of listquote and caseless.
Because includer removes relevant import statements (with certain caveats - see below) I only need to maintain one version.
configobj.py has the import statements and will import from listquote and caseless. 
includer takes this file, includes from caseless.py and listquote.py, removing the now unneeded import statements.
This produces fullconfigobj.py which has the external dependencies included.

It works by finding include commands that the python interpreter treats as comments.
Whitespace is ignored and module names/paths can be quoted (but need not be).
Relative paths will be resolved relative to the script they are in.
## include mymodule.py
##include c:\Py modules\my module.py
## INCLUDE "../Python Stuff/my module.py"

You can add a directory to the search path using the INCDIR command.
Relative paths will be resolved relative to the script they are in and turned into an absolute path.
##INCDIR c:\Python Stuff
##INCDIR '../Python Stuff'
This directory (the absolute path) remains in the search path for all other scripts that are included - be careful to avoid module name conflicts.

The INCLUDE command is designed to replace import commands of the form :
from listquote import *
from listquote import lineparse, elem_quote

Any 'import from..' statements relating to included modules will be removed.
Note that including a script is always the same as 'from module import *'.
Avoiding name clashes is up to you !

The INCLUDE command should be basically in the same place as the import statement is/would be.
Obviously trying to use module components before including them doesn't work.

In any 'included' script, any indented lines beneath an :
if __name__ == '__main__':
line will be removed. (Only if unindented though - or it's assumed to be part of program logic).

This program reads through a script.
It recognises and follows (also reads) all the #include lines.
It also keeps a record of all imported modules so far.
It deletes all import references to 'included' modules which are of the form 'from module import....'
It keeps a record of import statements - so it can go back and delete them if a file is included later.
If there is a straight import reference to a module it raises an error. 
(Because the code expects to reference module resources using the module name - the module is being included and so in normal circumstances won't be available to import). 

If you run it without arguments and the subdirectory 'tests/' exists... it will attempt to run the test case...


ISSUES
Any modules being imported/included with the same module name (in different scripts from different locations) will cause various problems. (all import statements removed, only one of the scripts included)
deleting modules and reimporting may cause problems
having an import statement for a module and using it before the include statement is bad
Bad indentation will confuse us
Won't deal/cope with non-ascii encoded scripts - except by mistake !
Syntax errors in 'import' statements might kill us
'from .. import .. as.....' won't work !
Multiple includes of the same file will be ignored.
Where we go back and remove a 'from .. import' line - we leave a blank line in it's place
All comments, docstrings etc from a file will also be included
"if __name__ == '__main__':" lines within """ """ will be picked up as real !!
Will pick up ##include lines (etc) from within """ """ (and include the files into the comment).
You can't yet put comments on the same line as a command..... (nor is their proper unquoting).
Because we remove lines like 'from module import...' this could lead to syntax errors.
e.g.
if test:
    from module import stuff
else:
    pass

would become :-
if test:
else:
    pass
which of course is a syntax error - could maybe insert a 'pass' ? Probably *very* rare and ought to get picked up at the first test...


TODO
Command line/config file stuff
Verbosity levels/logging using StandOut
GUI
Add a line at the start of the first file ? (File built with includer.py....)
Implement incstart/incstop commands
Allow the use of '#include module' - without the '.py' ?
Check and deal with indented import statements.
Add line numbers to error messages ?
Make errors non-fatal as an option
Conditional includes ? DEF and IFDEF ?? (I'm not a C programmer - so I've never used them...)
Use sys.path for search path ?
implement incstart/incstop command - also possibly commands for having several 'named' parts of a file and allow only importing parts....


CHANGELOG
13-08-02        Version 1.0.0
Well, it apepars to work.