void

Background

Console and GUI subsystems

Each EXE file has a flag that marks it either as GUI or CONSOLE subsystem application (there are other subsystems but they are not important for this article). Apps marked as console or GUI behave differently.

Console apps:

• When run as is, create a new console window for itself.

• When run from another console app (such as cmd.exe), inherit its console window and do not return until they're done.

GUI apps:

• When run as is, are invisible (until they create some GUI windows manually).

• When run from a console app (such as cmd.exe), return the control immediately and work in the background.

Nevertheless once started, they are identical. They can call the same functions, they have the same rights, and so far as Windows is concerned, each can be reconfigured to be another.

There's no principal difference between Console and GUI app, once they're running.

Consoles

Each process can have up to one attached console. Consoles are windows where you type the input and where the output is printed. They are created automatically by the system when a Console-marked application starts and has no inherited console.

Each console provides three standard handles, for input, output and error output. They are passed to the application as described below. When you write to an output/error handle, that data is displayed in the console. When you read from an input handle, the console lets the user type some characters and returns that.

Attaching a console means you can use those handles. Without a console attached even if you somehow get those handles, they will not work.

Attaching goes deeper than that: once the console window is closed, all attached processes will be terminated and there's no way to stop that, though you can receive a notification and be allowed to finalize.

When a console-marked app starts another console-marked app, the latter inherits the console. No new window is created, and it receives the same input, output and error handles.

GUI-marked app never inherit the console, no matter how they are started. But they still inherit the handles, which will be broken (see below).

By default in modern systems console window is implemented by conhost.exe. But there are apps which replace it when installed.

Attaching to a console

Any application, including a GUI-marked one, can attach to a console with AttachConsole(), by passing the process ID of a process with a console (copying a console from it).

If two applications are attached to the same console, both can read and write to it. Unless they coordinate, they'll produce a mess.

Any application can create a new console for itself with AllocConsole(). This will spawn a new console window, identical to the one Windows creates automatically for console-marked apps.

In fact, for console-marked apps Windows just calls it automatically, if no console is inherited.

Detaching from a console

Any application, including a console-marked one, can detach from a console with FreeConsole(). In that case standard handles stop working.

Yes, that means a console app can stop being a console app, if it wishes so.

Console windows are destroyed automatically once no one is attached to it. If you're the only one attached (because you created the console with AllocConsole() or because it was created for your console-marked process automatically), once you FreeConsole() the window disappears and you continue running.

If you inherited the console from a parent process or attached to it manually, it'll stay for that other process.

Standard handles

To input and output data in console apps Windows provides three read-write handles which can be obtained from GetStdHandle(): STD_INPUT_HANDLE, STD_OUTPUT_HANDLE and STD_ERROR_HANDLE.

These are standard Windows handles, same as returned by CreateFile().

For a console app, all of these point to an attached console. If the input is redirected, they will point to files or to pipes to another app. For a GUI app started from Explorer, all will be zero.

By default these handles are inherited from parent to child. If you start a process from console, it will have its standard handles set to that console.

There are two ways to override handles for another app:

• When starting an app with CreateProcess, you can provide standard handles directly in a special structure.

• Or you can SetStdHandle() for your own process temporarily and start a child process like that (since standard handles are inherited).

To redirect input/output, the usual approach is to create an inheritable pipe, pass one end to a child process as an input and write to another. That's more or less what cmd.exe does when it chains applications. But you can pass any handles which support reading/writing, including network handles and file handles.

How console-marked apps are started

All processes under Windows are ultimately started with a variation of CreateProcess(). This function returns immediately once the process is started.

How is it, then, that when you run a console-marked app from a command line (cmd.exe) it does not return until it is finished?

The answer is that cmd.exe itself does that. It starts the process and if it notices that the process is a console-marked one, it waits for it to return, and sets ERRORLEVEL to its exit code.

Why does it do that? Because child command-line processes inherit the console and start writing to it, and if cmd.exe continued to use the console at the same time, the result would be a mess. So it waits until the console is free again.

In other words it's not built into the OS. If your command-line app starts another command-line app, by default it'll still return immediately, and that app will write to your console in parallel with you.

What if I don't want to share my console?

You can pass a flag CREATE_NEW_CONSOLE or DETACHED_PROCESS to attach a new console to a console-marked app, or to start it without any (like a GUI app!)

Can I share my console with a GUI app?

By default, no. There's no way to start a GUI-marked app with inherited console. There's no way to attach a console to another application. Only the application itself can call AttachConsole to attach one.

Can I start a GUI app with an attached console?

Yes, but only with a new one (pass CREATE_NEW_CONSOLE).

Reading and writing to console from GUI app

Since standard handles are inherited, GUI apps run from console receive handles to that console. But those handles will not work, because the console itself is not attached.

To make them work, it's enough to do AttachConsole(ATTACH_PARENT_PROCESS).

There's a host of cases that needs to be covered:

• Standard handles (from GetStdHandle()) may be 0, which indicates that either you were started from Explorer, or that whoever started you wanted you to throw away your output. In this case, you do not have to AttachConsole().

• Standard handles may be redirected to a file, or a pipe, or elsewhere. In which case you still do not have to AttachConsole(). You should just write where pointed. How to check what those handles are? GetFileType() == FILE_TYPE_CHAR, but it wouldn't work — see below.

• Standard handles may look like a console handles, but come from your own console, attached to you automatically because someone started you with CREATE_NEW_CONSOLE. It would not be nice if you detached from it and attached to a parent one, so you must not call AttachConsole in this case. How to tell if you have your own console? GetConsoleWindow().

• Standard handles may come from a console, but an entirely different one because someone screwed up. In which case there's nothing you can do and the behavior is undefined, but it would probably be better to just write wherever pointed, and not AttachConsole (since parent is unrelated).

Summarizing, what do we do?

If we have a console (GetConsoleWindow() <> 0), do nothing! We have explicitly been told exactly where to write.

If all the handles are zero, and you cared to check, do nothing! Either we don't have a console-enabled parent (nowhere to attach), or been explicitly told to be silent.

Otherwise we could check GetFileType() to see if any of our handles is FILE_TYPE_CHAR. But there are two troubles with that. First, FILE_TYPE_CHAR also means stuff like LPT printers and not only consoles. And second, while the console is not attached, console handles will return FILE_TYPE_UNKNOWN.

So there's no clear way to tell. As an optimization, you could check whether all of the handles are FILE_TYPE_PIPE and FILE_TYPE_DISK and in this case skip the AttachConsole(), but there's plenty of cases where you don't need AttachConsole() and can't tell it.

So just do it. AttachConsole() returns FALSE if there's no console underneath, and has no other consequences; we've already determined we have no other console to lose, so even if we attach to a parent one for no reason, it won't hurt. Or will it?

What are the consequences of attaching to a parent console?

There are two:

• Your process will be terminated if a console closes (see the section on Consoles)

• Parts of your code which write to console may write to the parent console unexpectedly for the parent process.

Remember that for GUI-marked apps, cmd.exe returns immediately and the user may be typing another command, as you steal the console from them and output your messages. Therefore attaching to a console must be accompanied by a parent application waiting until you finish (how to achieve that will be described in the sections below).

Since there's no way to guarantee that you're being run in that fashion, it may be sensible to not attach to a console by default. Only if you're passed a flag (say, "/console") indicating that the user (or an intermediate app) wants to run you in a console mode, only then should you attach a parent console.

How to run a console-attaching program

By default, cmd.exe does not wait for your process to return. It returns the control to the user immediately. If you attach the console and start writing to it, you'll be interfering with user's actions.

Even if you do not attach the console, your error code will not be delivered to the cmd.exe, and will not be put into ERRORLEVEL, as expected by scripts.

There are two approaches:

• start "" /WAIT will start the app and wait for it to return. It will also properly handle the error code. This is entirely sufficient if your app does AttachConsole() and it works (see below about problems with runtimes).

• You can write your own console-marked wrapper to start the app, wait for it to finish, then query and return it's error code. This is preferable if you want a neat console shortcut to your GUI app, and when more complicated handling is needed (see below).

Consoles, standard handles and runtimes

AttachConsole() is fully enough to make ReadFile() and WriteFile() on console handles work just like they would in a console app.

But in a real world, most people write to console through some kind of language abstraction (printf() in C++, writeln() in Pascal and so on).

These abstractions are wrappers around platform-dependent functions, and on Windows resolve to querying GetStdHandle(STD_OUTPUT_HANDLE) and doing WriteFile() to it. But they introduce problems.

There's two kinds of problems I've encountered:

• Buffering, and

• Handle caching

Buffering

Most runtimes do not write to files immediately as you call the equivalent of printf("text"). Badly written code often outputs data char by char, and it would've been too many system calls. Instead, they collect the data locally and push it to the operating system only occasionally.

C/C++ for example has three modes of buffering available for CRT FILEs: no buffering (output immediately), line buffering (output on each CR/LF) and block buffering (output when something like 2048/4096 bytes are collected).

CRT tries to be smart about it and chooses different buffering schemes for different files. For stdout it chooses line buffering, as it assumes we want to see each line as soon as it's printed, not ten minutes later when enough data has been collected.

But when it detects the output is redirected (i.e. standard handles point to something other than FILE_TYPE_CHAR), it switches stdout/stderr to block buffering.

For instance, if you run a C++ app with it's output redirected to a pipe, and if you read from that pipe and put that on the screen, you will not see the output line by line as the program executes. You will instead receive it in blocks of several lines, which is not how we expect console apps to function.

Similar buffering exists in most runtimes, although sometimes it's smaller and less noticeable.

In C Runtime it can be disabled by calling setvbuf(stdout, NULL, _IOLBF, BUFSIZ) to enable line mode, or setbuf(stdout, NULL) to disable buffering at all. It needs to be done manually even if you output to console to which you attach with AttachConsole, because the buffering is decided before entering main(), and at that time console is not yet attached, so GetFileType() returns FILETYPE_UNKNOWN.

In particular, in Visual C Runtime line buffering is not implemented at all, so setbuf(stdout, NULL) should be used instead.

Other runtimes have their own ways of disabling buffering. If there's none, sometimes you'll have to bear with it.

Handle caching

Some runtimes, notoriously Visual C Runtime, inspect standard handles once and then remember the results forever. If your application starts with one set of standard handles, and then you use SetStdHandle to redirect your own output, that redirect will not be applied to stdout because it has already decided where to write.

This is especially bad with ATTACH_PARENT_CONSOLE method. By the time you get to do AttachConsole, C Runtime has already inspected the standard handles, called GetFileType() on them, seen that they're FILE_UNKNOWN_TYPE (because their console wasn't attached) and marked to never write anything to stdout/stderr.

Is there anything you can do? No.

There's no way to make C Runtime re-decide. You can open a new FILE for a standard handle with _open_osfhandle, but you cannot assign it to stdout for it is a constant. You can freopen stdout, but only to a file name, not to a handle already opened. Finally, some sources teach to freopen("CON"), but this will always return a handle to attached console (no matter if it's not set as your output), what if your standard handle is actually redirected to a file? Or a pipe?

So in short, there's no reliable, non-hackish way (which doesn't involve patching internal CRT structures).

In this case ATTACH_PARENT_CONSOLE method is not going to work, and you're restricted to a console-marked wrapper with pipes (see in Solutions).

Compiled solutions

Summarizing all the knowledge we've accumulated, we'll list approaches to making a GUI application behave in a console-like way.

ATTACH_PARENT_CONSOLE

When our app detects its parent process has a console, and it doesn't have a console of its own, and perhaps it's started with a "/console" flag, it calls AttachConsole(ATTACH_PARENT_CONSOLE) and ignores the result.

If used with a "/console" flag, it then can adjust buffers and other settings as expected from a console app.

if ((GetConsoleWindowHandle() == NULL) && (haveArg("/console"))) {
 AttachConsole(ATTACH_PARENT_CONSOLE);
 setbuf(stdout, NULL);
 setbuf(stderr, NULL);
}
It can then be run with start "" /WAIT app.exe params.
Console wrapper
A small console application can be written which starts the target application, passes it's entire command line unchanged, waits for it to return and returns the error code.
Usually there's not even a need to change the GetCommandLine() before passing it as lpCommandLine to the CreateProcess(), as the first argument in it needs not to be equal to the actual executable name. It's enough to pass a correct new executable into the lpApplicationName.
Such a wrapper may include any additional flags you wish, for instance it may pass "/console" flag to your app to make it adjust its behavior (see ATTACH_PARENT_CONSOLE).
You do not need to override any of the standard handles, as they'll be automatically inherited.
Console wrapper with pipe redirection
When ATTACH_PARENT_CONSOLE method doesn't work, you'll have to write a wrapper with pipe redirection. Do the same as in the previous step, only create three pipes in addition.
These pipes must be created with "Inheritable" flag set.
SECURITY_ATTRIBUTES sa;
memset(&sa, 0, sizeof(sa));
sa.nLength = sizeof(sa);
sa.lpSecurityDescriptor = NULL;
sa.bInheritHandle = TRUE;
CreatePipe(out, in, &sa, 0));
For each pipe, pass one side as a standard handle to the child process and close it after the CreateProcess() call, keep the other side and read or write to it (make sure to keep the side you need, one is for reading, the other for writing).
Start a process with inherit handles flag set to true, then wait for it to finish, meanwhile translating input to it and reading and posting output. You'll probably need a separate thread for input (readln() and write to input pipe in cycle), and you can handle both output and error output pipes in the main thread, by looking with PeekNamedPipe() and reading if there's something to read.
This way when the target app starts, it already has valid input/output redirected handles. You may need cooperation of the target app to configure buffers and such like in ATTACH_PARENT_CONSOLE method.
Controlling the lifetime of a child process
Since your wrapper is a console-marked application, it is attached to a parent console and will be terminated if you press Ctrl-C, or if a parent console is closed.
But your child process is a GUI app. If it does not do ATTACH_PARENT_CONSOLE, it will remain in the background even after you're terminated.
To work around that, set your own SetConsoleCtrlHandler(). This function will receive notifications when Ctrl-C is pressed or the console you're attached to is about to go down.
You will not be able to prevent that, but you'll be able to terminate the child process with you.
tee
There's a quick and dirty way to test redirection before writing a wrapper. Use tee, an utility from unix-utils, available from cygwin. It's purpose is to redirect the output both to file and to the screen; if you omit the file, you're just getting the redirection wrapper:
MyGUIapp.exe | tee
Console app + GUI wrapper
Another way of having a dual console/GUI app is to mark it as console and FreeConsole immediately after start if you're running in a GUI mode. But that still produces flickering console window (ugly).
But why, you can make a thin wrapper. It's only role is to start the console app with DETACHED_PROCESS flag. This flag instructs Windows to not create a console window. Your app can then detect that it has no attached console (with GetConsoleWindowHandle) and run in a GUI mode.
This is the simplest approach of all, and with the least amount of code.
FAQ / TLDR
Yes, you can have a chameleon console/GUI app, but reliably only through a secondary helper app.
Can I write to console from my GUI app, if it's run from one?
Yes, so long as you AttachConsole(ATTACH_PARENT_CONSOLE) or AllocConsole() and use WinAPI functions. But you don't want to do this.
Why don't I want to do that?
Because the parent application does not expect this. You should only ATTACH_PARENT_CONSOLE if you're run in a certain way.
How should I be run so that I can attach parent console normally?
By start "" /WAIT appname.exe or through a special wrapper app.
How do I know that I'm run like that?
You don't. The usual way is to pass you a flag, something like "/console". But you may decide to be reckless and just always attach the console.
Will my default printfs/writelns work everywhere in a program?
Sometimes. Some runtimes permit this, others don't. C Runtime doesn't, Delphi does. If your runtime doesn't, WriteFile will work, but printfs won't.
Can't I do something so that my normal printfs work?
You can make a console wrapper around your app which runs it with redirected handles. It's more work, but it will be almost transparent to your GUI app.
Can my GUI app tell if it has a console?
Yes, by checking GetConsoleWindowHandle(). But most of the time, your GUI app will NOT have a console. It does not inherit the console by default, even if started from one. You have to connect to one manually.
Okay, can my GUI app tell if there's a parent console to connect to?
It probably can, but it's easier to just go ahead and try to connect. If it fails, it fails. But make sure you don't have your own console beforehand.
Can I make some existing GUI app (pre-compiled) write to console? (I know it calls printfs, even though it has no console)
Yes you can, by writing a wrapper. But there may be inconveniences (buffered output). One simple way to test it is to use tee from unix-utils.
Further reading
 
How do I write a program that can be run either as a console or a GUI application? — The Old New Thing
 
How to make an application as both GUI and console application (more options)
 
Creating a Child Process with Redirected Input and Output — MSDN
 
Consoles — MSDN