Red Hat

Running an Embedded WildFly Host Controller in the CLI

WildFly supports running a standalone server in Offline mode as an embedded server. An analogous capability has been added for domain mode in the form of running an offline host controller. This article discusses some of the details and possible applications of this feature.

Introduction

For those not familiar with the existing embedded standalone server, see: Offline mode Running an Embedded WildFly 9 Server in the CLI · WildFly. One of the major use cases of this feature was to allow local administration of a WIldFly / JBoss EAP instance without requiring a socket based connection, or opening any local ports but still enabling configuration and administration operations. The embedded approach also allows for a fast boot that will allow multiple concurrent running instances to co-exist without configuring any additional socket offsets or interfaces to avoid port conflicts etc.

Using jboss-cli.sh and the embed-host-controller command enables a fast and flexible means of configuring host controller instances and servers.

Embedded Host Controller

In a similar way to the standalone embed-server command, the embed-host-controller command is provided, to start an embedded host controller running in the CLI. This host controller is started in admin-only mode, and no servers are started. [See below for more details on admin-only restrictions].

Starting an embedded Host Controller:

$ ./bin/jboss-cli.sh

You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-host-controller --std-out=echo
11:07:46,723 INFO  [org.jboss.modules] (AeshProcess: 1) JBoss Modules version 1.6.0.Final
11:07:46,825 INFO  [org.jboss.msc] (AeshProcess: 1) JBoss MSC version 1.2.7.SP1
11:07:46,870 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: WildFly Full 11.0.0 (WildFly Core 3.0.1.Final) starting
[ Note: some startup output omitted for brevity. ]
11:07:48,845 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 11.0.0 (WildFly Core 3.0.1.Final) (Host Controller) started in 2105ms - Started 56 of 61 services (18 services are lazy, passive or on-demand)

[domain@embedded /] ls -l
ATTRIBUTE                VALUE           TYPE
domain-organization      undefined       STRING
launch-type              EMBEDDED        STRING
local-host-name          master          STRING
management-major-version 5               INT
management-micro-version 0               INT
management-minor-version 0               INT
name                     Unnamed Domain  STRING
namespaces               []              OBJECT
process-type             Host Controller STRING
product-name             WildFly Full    STRING
product-version          11.0.0          STRING
release-codename         Kenny           STRING
release-version          3.0.1.Final     STRING
schema-locations         []              OBJECT

CHILD                     MIN-OCCURS MAX-OCCURS
core-service              n/a        n/a
deployment                n/a        n/a
deployment-overlay        n/a        n/a
extension                 n/a        n/a
host                      n/a        n/a
host-exclude              n/a        n/a
interface                 n/a        n/a
management-client-content n/a        n/a
path                      n/a        n/a
profile                   n/a        n/a
server-group              n/a        n/a
socket-binding-group      n/a        n/a
system-property           n/a        n/a

The non-modular client may also be used from the bin/client WildFly distribution:

$ java -jar bin/client/jboss-cli-client.jar
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-host-controller --jboss-home=/wildfly-11.0.0
Warning! The CLI is running in a non-modular environment and cannot load commands from management extensions.
[domain@embedded /] cd /host=master/server-config=server-one
[domain@embedded server-config=server-one] ls -l
ATTRIBUTE                            VALUE             TYPE
auto-start                           true              BOOLEAN
cpu-affinity                         undefined         STRING
group                                main-server-group STRING
name                                 server-one        STRING
priority                             undefined         INT
socket-binding-default-interface     undefined         STRING
socket-binding-group                 undefined         STRING
socket-binding-port-offset           0                 INT
status                               STOPPED           STRING
update-auto-start-with-server-status false             BOOLEAN

CHILD           MIN-OCCURS MAX-OCCURS
interface       n/a        n/a
jvm             n/a        n/a
path            n/a        n/a
ssl             n/a        n/a
system-property n/a        n/a
[domain@embedded server-config=server-one]

See Modular vs Non-Modular Classloading and JBOSS_HOME in the original embedded server news article for details.

Executing commands:

[domain@embedded /]  /host=master/interface=public:write-attribute(name=inet-address, value=127.0.0.1)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined,
    "response-headers" => {"process-state" => "reload-required"}
}

Reloading:

[domain@embedded /] reload --host=master --admin-only=true
11:17:20,092 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 11.0.0 (WildFly Core 3.0.1.Final) (Host Controller) started in 279ms - Started 56 of 61 services (18 services are lazy, passive or on-demand)
[domain@embedded /]

Note that currently, the embedded host controller may only be started / reloaded in admin-only mode. In domain mode, servers are started and stopped via use of the process controller which is not currently supported as part of embedded mode.

Stopping:

[domain@embedded /] stop-embedded-host-controller
11:08:29,925 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: WildFly Full 11.0.0 (WildFly Core 3.0.1.Final) stopped in 13ms

Command usage:

The embed-host-controller command has several options that behave in the same way as the previously mentioned standalone embedded server, which will not be discussed again here. The relevant embed-host-controller parameters are:

-c                - Name of the domain configuration file to use
                     (default is "domain.xml")
                     (Same as --domain-config)

--domain-config   - Name of the domain configuration file to use
                     (default is "domain.xml")
                     (Same as -c)

--host-config     - Name of the host configuration file to use
                     (default is "host.xml")

As mentioned above, --jboss-home, --std-out and --timeout may also be provided and function in the same manner as the embed-server command. The configuration files mentioned above (domain.xml, host.xml) above should be located in the $JBOSS_HOME/domain/configuration directory (or under the location pointed to by the system property jboss.domain.config.dir.) [See Command Line Properties for additional details on those properties.]

For example, to start an embedded host controller with configuration files contained in the otherdomain/configuration directory:

[wildfly-11]$ ./bin/jboss-cli.sh -Djboss.domain.config.dir=/wildfly-11/otherdomain/configuration
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-host-controller --std-out=echo
11:26:44,122 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 11.0.0 (WildFly Core 3.0.1.Final) (Host Controller) started in 1894ms - Started 56 of 61 services (18 services are lazy, passive or on-demand)
[domain@embedded /]

Configuration may then proceed and will be persisted to the otherdomain/configuration directory. This directory must already exist and contain base copies of the required configuration files (host.xml, domain.xml etc.).

Scripted configuration

The embedded host controller may be useful for configuration from a prepared file of scripted CLI commands. For example:

$ cat commands.cli
embed-host-controller
/server-group=main-server-group:write-attribute(name=socket-binding-port-offset, value=100)
/host=master/server-config=server-one:write-attribute(name=auto-start, value=false)
deploy --all-server-groups test.war
stop-embedded-host-controller

$ ./bin/jboss-cli.sh --file=commands.cli
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}

This approach may be used for a variety of setup and configuration tasks, for example setting up unit or integration tests quickly using the embedded host controller, then restarting in domain mode using domain.sh may require less time than starting the host controller normally using domain.sh, performing configuration and deployment etc, then restarting.

Other examples

Set server socket-binding-port-offset

In order to allow more than one running instance on the same host, a common configuration for testing (or any scenario needing to run a domain controller and a slave host controller (with servers) on the same host), a socket-binding-port-offset is commonly used. The slave host is configured to have a port offset so that the ports already in use by the domain controller’s servers do not conflict with those of the slave.

[domain@embedded /] /server-group=main-server-group:write-attribute(name=socket-binding-port-offset, value=100)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}

Configure connection to remote domain controller

When configuring a slave host controller, configure the connection to the domain controller.

[domain@embedded /] /host=master:write-remote-domain-controller(host=remotedc.somedomain.tld, security-realm=ManagementRealm)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined,
    "response-headers" => {"process-state" => "reload-required"}
}

System property

This can be useful as an initial configuration step before the host controller is started with domain.sh:

[domain@embedded /] /server-group=main-server-group/system-property=foo:add(value=bar)
{
    "outcome" => "success",
    "result" => undefined,
    "server-groups" => undefined
}

Future Direction

In the future we’d like to allow for starting the embedded host controller with some additional features, such as empty configurations in host and domain configuration files (similar to standalone embedded), and also re-examine the meaning and usage of --admin-only in the context of the embedded host controller.

Running an Embedded WildFly 9 Server in the CLI

In WildFly 9 Beta1 we are introducing a new capability to our Command Line Interface (CLI) tool: the ability to embed a WildFly standalone server process inside the CLI process, with the CLI interacting with the embedded server in a manner consistent with how it interacts with a remote WildFly server. All of the standard CLI commands that you can use to administer a remote server are available.

The immediate goal is to support direct local administration of a WildFly installation via the CLI without requiring a socket-based connection. The general use case is initial setup type activities where the user doesn’t want to have to launch a WildFly server and have it be visible on the network. RBAC configuration and other management security configuration is one use case; another is a desire users have expressed to be able to do configuration without first having to edit any xml to avoid port conflicts on 9990 or 9999.

Longer term, since the CLI is itself embeddable, this opens up the possibility of embedding the CLI inside some other process (say a test class, a provisioning tool or a jar with a main) and then launching the embedded server and using the embedded CLI as a convenient management tool. For example, an installer or other provisioning tool could include in its own configuration a small amount of CLI script, which the tool would use to customize a stock WildFly configuration, say by adding a datasource configuration. The user of the tool would only need to understand the CLI script.

Simple Example

Here I start the CLI in a modular environment, launch an embedded server, do a couple simple CLI things, and stop the embedded server:

$ bin/jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-server --std-out=echo
12:10:15,300 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.1.Final
12:10:15,983 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.4.Final
12:10:16,049 INFO  [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) starting
12:10:16,891 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute enabled is deprecated, and it might be removed in future version!
12:10:17,055 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) started in 7113ms - Started 35 of 48 services (19 services are lazy, passive or on-demand)
[standalone@embedded /] ls -l
ATTRIBUTE                VALUE                  TYPE
launch-type              EMBEDDED               STRING
management-major-version 3                      INT
management-micro-version 0                      INT
management-minor-version 0                      INT
name                     taozi                  STRING
namespaces               []                     OBJECT
process-type             Server                 STRING
product-name             WildFly Full           STRING
product-version          9.0.0.Alpha2-SNAPSHOT  STRING
profile-name             undefined              STRING
release-codename         Kenny                  STRING
release-version          1.0.0.Alpha18-SNAPSHOT STRING
running-mode             ADMIN_ONLY             STRING
schema-locations         []                     OBJECT
server-state             running                STRING
suspend-state            RUNNING                STRING

CHILD                MIN-OCCURS MAX-OCCURS
core-service         n/a        n/a
deployment           n/a        n/a          brew unlink gcc-4.2
deployment-overlay   n/a        n/a
extension            n/a        n/a
interface            n/a        n/a
path                 n/a        n/a
socket-binding-group n/a        n/a
subsystem            n/a        n/a
system-property      n/a        n/a
[standalone@embedded /] cd socket-binding-group=standard-sockets/socket-binding=management-http
[standalone@embedded socket-binding=management-http] :write-attribute(name=port,value=19990)
{"outcome" => "success"}
[standalone@embedded socket-binding=management-http] reload --admin-only=false
12:12:34,615 INFO  [org.jboss.as] (MSC service thread 1-16) WFLYSRV0050: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Beta1) stopped in 16ms
12:12:34,621 INFO  [org.jboss.as] (MSC service thread 1-16) WFLYSRV0049: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Beta1) starting
. . . .
12:12:36,176 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Beta1 (WildFly Core 1.0.0.Beta1) started in 1505ms - Started 202 of 379 services (210 services are lazy, passive or on-demand)
[standalone@embedded socket-binding=management-http] stop-embedded-server
12:12:43,352 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
12:12:43,352 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
12:12:43,364 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-13) WFLYJCA0019: Stopped Driver service with driver-name = h2
12:12:43,368 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
12:12:43,380 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to /127.0.0.1:8080
12:12:43,384 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-13) WFLYUT0004: Undertow 1.2.0.Beta8 stopping
12:12:43,393 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: WildFly Full 9.0.0.Beta1 (WildFly Core 1.0.0.Beta1) stopped in 13ms
[disconnected socket-binding=management-http] quit

Here I do a similar thing, but using the non-modular jboss-cli-client.jar included in the bin/client directory of the WildFly distribution. I also don’t include the --std-out=echo param, so I don’t see the server logging in the console:

$ java -jar bin/client/jboss-cli-client.jar
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-server --jboss-home=/home/brian/dev/wildfly/dist/target/wildfly-9.0.0.Alpha2-SNAPSHOT
[standalone@embedded /] cd socket-binding-group=standard-sockets/socket-binding=management-http
[standalone@embedded socket-binding=management-http] :read-attribute(name=port)
{
    "outcome" => "success",
    "result" => 19990
}
[standalone@embedded socket-binding=management-http] quit

I’ll explain the --jboss-home parameter below. See Modular vs Non-Modular Classloading and JBOSS_HOME.

Finer points

Specifying the Server Configuration

The embed-server command supports -c and --server-config parameters that can be used to specify the name of the configuration file to use. If not specified, the default is standalone.xml.

[disconnected /] embed-server --server-config=standalone-full.xml

Starting with an Empty Configuration

Embedding the server opens up the possibility of starting from a completely empty, even non-existent, configuration. Such a configuration makes no sense without embedding, as no configuration means no remote management interface, and thus no means to change the configuration to something useful. But a server embedded in the CLI tool doesn’t have that problem.

So, the embed-server command allows you to specify that an initially empty configuration should be used:

[disconnected /] embed-server --server-config=my-config.xml --empty-config

That command will fail if file $JBOSS_HOME/standalone/configuration/my-config.xml already exists. This is to avoid accidental deletion of a configuration file. However, you can specify that you want any existing configuration removed:

[disconnected /] embed-server --server-config=my-config.xml --empty-config --remove-existing

Here I launch the CLI telling it to run a CLI script (available here) that starts an embedded server with an empty configuration and then uses CLI commands to build the entire server configuration. The resulting configuration is equivalent to the standard standalone-full-ha.xml config that ships with WildFly:

$ bin/jboss-cli.sh --file=/home/bstansberry/tmp/embedded-server.txt
The batch executed successfully
The batch executed successfully
process-state: reload-required
$

The script is long but conceptually straightforward. First it launches the embedded server:

embed-server --server-config=standalone-empty.xml --empty-config --remove-existing

Then it runs a CLI batch to add all the desired extensions:

# Extensions first
batch
/extension=org.jboss.as.clustering.infinispan:add
/extension=org.jboss.as.clustering.jgroups:add
/extension=org.jboss.as.connector:add
. . . .
/extension=org.wildfly.extension.undertow:add
/extension=org.wildfly.iiop-openjdk:add
run-batch

Once this batch runs, the server will understand the management APIs exposed by those extensions, so the rest of the configuration can be applied. This is done in a second batch:

# Other
batch
/core-service=management/security-realm=ManagementRealm:add(map-groups-to-roles=false)
. . . .
/subsystem=webservices/client-config=Standard-Client-Config:add
/subsystem=weld:add
run-batch

It works!

$ bin/standalone.sh -c standalone-empty.xml
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /Users/bstansberry/dev/wildfly/wildfly/dist/target/wildfly-9.0.0.Alpha2-SNAPSHOT

  JAVA: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java

  JAVA_OPTS:  -server -XX:+UseCompressedOops  -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

12:24:45,565 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.1.Final
12:24:45,775 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.4.Final
12:24:45,843 INFO  [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: WildFly Full 9.0.0.Beta1 (WildFly Core 1.0.0.Beta1) starting
. . . .
12:24:48,649 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
12:24:48,649 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
12:24:48,649 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Beta1 (WildFly Core 1.0.0.Beta1) started in 3365ms - Started 246 of 478 services (281 services are lazy, passive or on-demand)

Admin-only Mode

By default the embedded server will be started in admin-only mode. This is because the main expected use cases are for initial configuration. A server running in admin-only mode will only start services related to server administration but will not start other services or accept end user requests.

This can be changed with a parameter to the embed-server command:

[disconnected /] embed-server --admin-only=false

Same as with a non-embedded server, a server can be moved in and out of admin-only using the CLI reload command:

[standalone@embedded /] reload --admin-only=false

Admin-only Mode and the Server’s Management Interfaces

One of the goals of this work is to support use cases where the server being configured is completely invisible on the network. Normally, the management interfaces themselves open sockets (e.g. port 9990, 9999), even when the server is in admin-only mode. But, what if there is a port conflict on those ports, with the purpose of using the offline CLI being to change settings to avoid the conflict?

To account for this, we have changed the behavior of the management interface resources. Now, if those resources detect they are running in an embedded server and the running mode is admin-only, the services for the remote management interfaces will not be started. The server will not be visible to remote management clients.

Controlling stdout

The CLI uses stdout heavily. The embedded server may also want to write to stdout, particularly for console logging. These two uses of stdout have the potential to interfere with each other, particularly in an interactive session where the CLI may output a command prompt and then the server logs something, resulting in the prompt being in the middle of server log messages, possibly in the middle of a line. The interactive CLI will still work if this happens, but it can be disorienting.

The embed-server command includes a parameter to allow the user to control what happens to output the embedded server writes to stdout:

  • --std-out=echo — the output from the server is allowed to go to the CLI’s stdout, allowing the user to see logging, but at the risk of mixing the CLI prompt with server logging

  • --std-out=discard — the output the server attempts to send to stdout is discarded. Users should look at the server.log file to see server logging.

The default behavior is --std-out=discard

Boot Timeout

By default, the embed-server command will block indefinitely waiting for the embedded server to reach server-state running; i.e. to complete boot. The amount of time to wait can be controlled by using the --timeout parameter

[disconnected /] embed-server --timeout=30

The value is in seconds.

A value of less than 1 means the embed-server command will not block waiting for boot to complete. Rather, it will return as soon as boot proceeds to the point where the internal ModelController service is available, allowing the CLI to obtain an internal client to use to execute management operations.

A server in admin-only mode would typically boot very quickly, so configuring this timeout would be more useful when --admin-only=false is used.

Stopping the Embedded Server

To stop an embedded server but continue with your CLI session, use the stop-embedded-server command:

[standalone@embedded /] stop-embedded-server
[disconnected /]

If you also want to exit the CLI session, you can simply use the standard quit command:

[standalone@embedded /] quit
$

The embedded server will be stopped cleanly.

When an embedded server is running, the CLI shutdown command usually used to stop a remote server is not available. The shutdown command has some behavior somewhat inconsistent with embedded server operation, so we chose to use a separate command for the embedded case.

Modular vs Non-Modular Classloading and JBOSS_HOME

As shown in the Simple Example section above, the CLI can either be running in a modular classloading environment (bin/jboss-cli.sh example) or in a flat classpath (java --jar bin/client/jboss-cli-client.jar example.) Either way, the embedded server runs in a modular classloading environment. There are some behavior differences between the two cases though:

  • If the CLI is running in a modular classloading environment:

    • the embedded server will use the same boot module loader as the CLI. The implication here is the CLI and server are running from the same WildFly installation, with the same module path and therefore the same set of modules available.

    • the embedded server will need to know where the root of the WildFly installation is. This must be provided to the CLI via the JBOSS_HOME environment variable. The jboss-cli.sh script sets this. If some other mechanism is used for starting the CLI, the JBOSS_HOME environment variable must be set.

  • If the CLI is not running in a modular classloading environment:

    • the embedding logic will set up an appropriate modular classloading environment for the server. The module path for the modular classloader will have a single element: <root_of_wildfly_installation>/modules

    • the embedded server will need to know where the root of the WildFly installation is. This must be provided to the CLI via one of the following mechanisms:

      • the JBOSS_HOME environment variable

      • the --jboss-home parameter to the embed-server command. If this is set, it takes precedence over any JBOSS_HOME environment variable

The --jboss-home parameter to the embed-server command is not supported in a modular CLI environment, as it would imply that the root of the embedded server could be something other than the root of the install from which the CLI is running.

Future Work

In WildFly 10 we’d like to also be able to embed a Host Controller process in the CLI, allowing similar offline configuration of WildFly managed domain hosts.

Enjoy!

back to top