Stealth Framework

Introduction

Stealth includes everything you need to build amazing chatbots with tools you know and love.

Assumptions

These docs are designed for intermediate Ruby developers who want to get started with the Stealth framework.

If you do not yet have experience with Ruby, we would recommend checking out these guides first:

What is Stealth?

Stealth is an open source Ruby framework for conversational voice and text chatbots.

Stealth is inspired by the Model-View-Controller (MVC) pattern. However, instead of calling them Views Stealth refers to them as Replies to better match the chatbot domain.

  • The Model layer represents your data model (such as Account, User, Quote, etc.) and encapsulates the business logic that is specific to your bot. By default, Stealth uses ActiveRecord.

  • The Controller layer is responsible for handling incoming requests from messaging platforms and providing and transmitting the response (reply).

  • The Reply layer is composed of “templates” that are responsible for constructing the respective response.

In addition to being inspired by Model-View-Controller (MVC) pattern, Stealth has a few other awesome things built in for you.

  • Plug and play services. Every service integration in Stealth is a Ruby gem. One bot can support multiple chat services (i.e. Facebook Messenger, SMS, Alexa, and more) and multiple NLP/NLU services.

  • Advanced tooling. From web servers to continuous integration testing, Stealth is built to take advantage of all the great work done by the web development community.

  • Hosting you know. Stealth is a Rack application. That means your bots can be deployed using familiar services like Docker and Heroku.

  • Ready for production. Stealth already powers bots for large, well-known brands including: TradeStation, Haven Life, BarkBox, and many others.

  • Open source. Stealth is MIT licensed to ensure you own your bot. More importantly, we welcome contributors to help make Stealth even better.

Getting Started

Ruby

Stealth has been tested on Ruby (MRI) 2.4.x and 2.5.x. While we don’t require any C-based Ruby gems, we haven’t yet certified Stealth on other VMs (such as JRuby).

Installation

You can install Stealth via RubyGems:

  gem install stealth

Next, you can create a new Stealth bot:

  stealth new <bot_name>

Local Development

Prerequisites

Stealth bundles Sidekiq in order to process background jobs. Therefore, it is required to run Redis in order to boot up a Stealth server.

Starting the Server

Once you have made your current working directory your Stealth bot, you can install gems:

bundle install

To boot your bot:

stealth server

You can also use stealth s. This will use the foreman gem to start the web server and Sidekiq processes together. Redis will have to be running for the server to start.

That’s it! You are now running Stealth.

Introspectable Tunnels to localhost

When developing locally, messaging services require access to your server in order to transmit user messages. We recommend downloading and using ngrok to create a local tunnel to your development machine.

  1. Download ngrok
  2. Start your Stealth server as detailed above.
  3. Open up an ngrok tunnel to your Stealth server and port (default 5000) like this: ngrok http 5000. ngrok will output a unique ngrok local tunnel URL to your machine.

When you provide your local ngrok URL to a messaging service, you will have to add /incoming/<service>. For example:

  • https://abc1234.ngrok.io/incoming/facebook
  • https://abc1234.ngrok.io/incoming/twilio

More details on service specific settings can be found on the GitHub page for each service gem.

The Basics

Directory Structure

When you open up your Stealth bot you will see the following file structure:

├── Gemfile
├── Procfile.dev
├── README.md
├── Rakefile
├── bot
│   ├── controllers
│   │   ├── bot_controller.rb
│   │   ├── catch_alls_controller.rb
│   │   ├── concerns
│   │   ├── goodbyes_controller.rb
│   │   └── hellos_controller.rb
│   ├── helpers
│   │   └── bot_helper.rb
│   ├── models
│   │   ├── bot_record.rb
│   │   └── concerns
│   └── replies
│       ├── catch_alls
│       │   └── level1.yml
│       ├── goodbyes
│       │   └── say_goodbye.yml
│       └── hellos
│           └── say_hello.yml
├── config
│   ├── boot.rb
│   ├── database.yml
│   ├── environment.rb
│   ├── flow_map.rb
│   ├── initializers
│   ├── puma.rb
│   ├── services.yml
│   └── sidekiq.yml
├── config.ru
└── db
    └── seeds.rb

Flows

A Flow is a general term to describe a complete interaction between a user and the bot. Flows are comprised of states, like a finite state machine.

For example, if a user was using your bot to receive an insurance quote, the flow might be named quote. Note: Stealth requires that flows be named in the singular form, like Rails.

A flow consists of the following components:

  1. A controller file, named in the plural form. For example, a quote flow would have a corresponding QuotesController.
  2. Replies. Each flow will have a directory in the replies directory in plural form. Again using the quote flow example, the directory would named quotes.
  3. An entry in config/flow_map.rb. The FlowMap file is where each flow and it’s respective states are defined for your bot.

Flows can be generated using a generator:

  stealth generate flow <NAME>

FlowMap

The FlowMap file is where each flow and it’s respective states are defined for your bot. Here is an example flow_map.rb:

class FlowMap

  include Stealth::Flow

  flow :hello do
    state :say_hello
    state :ask_name
    state :get_name, fails_to: :ask_name
  end

  flow :goodbye do
    state :say_goodbye
  end

  flow :catch_all do
    state :level1
    state :level2
  end

end

Here we have defined three flows: hello, goodbye, and catch_all. For the most part, these are default flows that are generated automatically when you create a new bot but we have made a few changes to highlight some functionality.

In the hello flow, the second state asks a user for their name. In the third state of hello, you see another option: fails_to. This is used to tell Stealth to return the user to the specified state if the get_name state raises an error or fails in another way. There are more details in the CatchAll section below.

Default Flows

When you generate a new Stealth bot, it comes packaged with three default flows. While you will likely add many flows of your own, we recommend keeping these three flows as they encourage good bot building practices.

Hello & Goodbye

These two flows make up the entrance and exit of your bot. We include blank examples on how to greet (say hello) and sendoff (say goodbye) your users. You can customize these flows to work with the design of your bot.

CatchAll

Stealth also comes packaged with a catch_all flow. Stealth CatchAlls are designed to handle scenarios in which the user says something the bot is not expecting or the bot encounters an error.

Error handling is one of the most important parts of building great bots. We recommend that bot designers and developers spend sufficient time building the CatchAll states.

See the Catch All (#catchalls) section for more information on how Stealth handles catch_all flows.

Say, Ask, Get

Stealth recommends you use the say, ask, and get prefix for your flow state names. It’s not required, but it is a convention we have found helpful to keep state names under control. It also helps other developers on your team follow along more easily.

SAY

SAY Stealth actions are for saying something to the user.

For example:

  def say_hello
    send_replies
  end

ASK

ASK Stealth actions are for asking something from the user.

For example:

  def ask_weather
    send_replies
    update_session_to state: 'get_weather_reponse'
  end

GET

GET Stealth actions are for getting and parsing a response from the user.

For example:

  def get_weather_reponse
    if current_message.message == 'Sunny'
      step_to state: "say_wear_sunglasses"
    elsif current_message.message == 'Raining'
      step_to state: "say_dont_forget_umbrella"
    end
  end

Sessions

A user of your bot can be in any single flow and state at any given moment. They can never be in more than one flow or state. For this reason, Stealth sessions are modeled using finite state machines.

Finite State Machines

Technically, each flow is its own state machine with its own states. Stealth, however, does not restrict the movement between states as rigidly. So while we find the state machine model helpful to learn sessions, don’t spend too much time on Wikipedia!

Redis Backed Sessions

User sessions are stored in Redis. Each session is a lightweight key-value pair. The key is the user’s ID from the service – so for Facebook it may be a long integer value: 100023838288224423. The value is the flow and state for the user separated by a “stabby” operator (->).

So if for example a user with ID 100023838288224423 is currently at the hello flow and ask_name state, the value for the key would be: hello->ask_name.

You likely won’t be interacting with sessions directly since Stealth manages it automatically for you. We just present it here for clarity into how sessions work.

Controllers

Controllers are responsible for handling incoming requests and getting a response back to the user (replies).

Naming Conventions

The controller’s methods, also referred to as actions, must be named after the flow’s states. So for example, given the flow:

flow :onboard do
  state :say_welcome
  state :ask_for_phone
  state :get_phone, fails_to: :ask_for_phone
end

The corresponding controller would be:

class OnboardsController < BotController
  def say_welcome

  end

  def ask_for_phone

  end

  def get_phone

  end
end

bot_controller.rb

Every Stealth project comes with a default bot_controller.rb:

class BotController < Stealth::Controller

  before_action :current_user

  def route
    if current_session.present?
      step_to session: current_session
    else
      step_to flow: 'hello', state: 'say_hello'
    end
  end

end

All of your controllers will inherit from BotController:

class QuotesController < BotController

end

Route Method

You can implement any method in BotController. Typically you will implement methods like current_user and methods for handling message payloads. The one method that BotController must implement is the route method.

The route method is called for every incoming message. In its default implementation, the route method checks whether the user already has a session, and if so routes them to that controller and action. If the user does not yet have a session, it will route them to the hello flow and say_hello action.

Stepping and Updating Sessions

Stealth provides a few built-in methods to help you route a user through your bot.

step_to

The step_to method is used to update the session and immediately move the user to the specified flow and/or state. step_to can accept a flow, a state, or both. step_to is often used after a say action where the next action typically doesn’t require user input.

Some examples of the different parameters:

step_to flow: 'hello' - Sets the session’s flow to hello and the state will be set to the first state in that flow (as defined by the flow_map.rb file). The corresponding controller action in the HellosController would also be called.

step_to state: 'say_hello' - Sets the session’s state to say_hello and keeps the flow the same. The say_hello controller action would also be called.

step_to flow: 'hello', state: 'say_hello' - Sets the session’s flow to hello and the state to say_hello. The say_hello controller action of the HellosController controller would also be called.

update_session_to

Similar to step_to, update_session_to is used to update the user’s session to a flow and/or state. It accepts the same parameters. However, update_session_to does not immediately call the respective controller action. update_session_to is typically used after an ask action where the next action is waiting for user input. So by asking a user for input, then updating the session, it ensures the response the user sends back can be handled by the get action.

Some examples of the different parameters:

update_session_to flow: 'quote' - Sets the session’s flow to quote and the state will be set to the first state in that flow (as defined by the flow_map.rb file).

update_session_to state: 'ask_zip_code' - Sets the session’s state to ask_zip_code and keeps the flow the same.

step_to flow: 'quote', state: 'ask_zip_code' - Sets the session’s flow to quote and the state to ask_zip_code.

send_replies

send_replies will instruct the Reply to construct the reply and transmit them. Not all of your controller actions will send replies. Typically in get action, you’ll get a user’s response, perform some action, and then send a user to a new state without replying.

The send_replies method does not take any parameters:

class ContactsController < BotController
  def say_contact_us
    send_replies
  end
end

This would render the reply contained in replies/contacts/say_contact_us.yml.

step_to_in

The step_to_in method is similar to step_to. The only difference is that instead of calling the respective controller action immediately, it calls it after a specified duration. It can also take a flow, state, or both.

For example:

step_to_in 8.hours, flow: 'hello', state: 'say_hello'

This will set the user’s session to the hello flow and say_hello state in 8 hours after being called. It will then immediately call that responsible controller action.

step_to_at

The step_to_at method is similar to step_to. The only difference is that instead of calling the respective controller action immediately, it calls it at a specific date and time. It can also take a flow, state, or both.

For example:

step_to_at DateTime.strptime("01/23/23 20:23", "%m/%d/%y %H:%M"), flow: 'hello', state: 'say_hello'

This will set the user’s session to the hello flow and say_hello state on Jan 23, 2023 @ 20:23. It will then immediately call that responsible controller action.

Available Data

Within each controller action, you have access to a few objects containing information about the session and the message the being processed.

current_session

The user’s session is available to you at anytime using current_session. This is a Stealth::Session object. It has a few notable methods:

flow_string: Returns the name of the flow. state_string: Returns the name of the state. current_session + 2.states: Returns a new session object 2 states after the current state. If we’ve passed the last state, the last state is returned. current_session - 2.states: Returns a new session object 2 states before the current state. If we’ve passed the first state, the first state is returned.

current_message

The current message being processed is available to you at anytime using current_message. This is a Stealth::ServiceMessage object. It has a few notable methods:

sender_id: The ID of the user sending the message. This will vary based on the service integration. timestamp: Ruby DateTime object of when the message was transmitted. service: String indicating the service from where the message originated (i.e., ‘facebook’). messsage: String of the message contents. payload: This will vary by service integration. location: This will vary by service integration. attachments: This will vary by service integration. referral: This will vary by service integration.

current_service

This will be a string indicating the service from where the message originated (i.e., ‘facebook’ or ‘twilio’)

has_location?

Returns true or false depending on whether or not the current_message contains location data.

has_attachments?

Returns true or false depending on whether or not the current_message contains attachments.

current_user_id

A convenience method for accessing the user’s ID.

Models

Models in Stealth are powered by ActiveRecord. Your bot may not need to persist data, but if it does, ActiveRecord comes built in. We’ve tried to keep things identical to Ruby on Rails.

ActiveRecord Models

An ActiveRecord model in Stealth inherits all of the functionality from ActiveRecord. An empty model looks like this in Stealth:

class Quote < BotRecord

end

With the exception of inheriting from BotRecord instead of ApplicationRecord, everything else matches what is in the ActiveRecord documentation.

Configuration

Configuring a database is done via config/database.yml. A sample database.yml file is included when you generate your bot. It is configured to use SQLite3. For more options please refer to the Ruby on Rails documentation.

Migrations

In order to use your models, you’ll need to generate migrations to create your database schema:

stealth g migration create_users

This will create a migration named CreateUsers. To migrate your database:

stealth db:migrate

For more information about migrations, seed data, creating databases, or dropping databases please refer to the Ruby on Rails documentation.

Just remember to prefix your commands with stealth rather than rails.

Replies

Stealth replies can send one or more replies to a user. Reply types are dependent on the specific messaging service you’re using. Each service integration will detail it’s supported reply types in it’s respective docs.

However, here is a generic reply using text, delays, and suggestions.

- reply_type: text
  text: "Hello. Welcome to our Bot."
- reply_type: delay
  duration: 2
- reply_type: text
  text: "We're here to help you learn more about something or another."
- reply_type: delay
  duration: 2
- reply_type: text
  text: 'By using the "Yes" and "No" buttons below, are you interested in do you want to continue?'
  suggestions:
    - text: "Yes"
    - text: "No"

Format

Stealth reply templates are written in YAML. Stealth doesn’t use advanced YAML features, but we do recommend you familiarize yourself with the syntax. In the above reply example, you should be able to see there are 5 replies included in the reply file.

Caveat: YAML interprets “yes”, “no”, “true”, “false”, “y”, “n”, etc (without quotes) as boolean values. So make sure you wrap them in quotes as we did above.

ERB

Reply templates currently support ERB:

- reply_type: text
  text: "Hello, <%= current_user.first_name %>. Welcome to our Bot."
- reply_type: delay
  duration: 2
- reply_type: text
  text: "We're here to help you learn more about something or another."
- reply_type: delay
  duration: 2
<% if current_user.valid? %>
  - reply_type: text
    text: 'By using the "Yes" and "No" buttons below, are you interested in do you want to continue?'
    suggestions:
      - text: "Yes"
      - text: "No"
<% end %>

With ERB in your reply templates, you can access controller instance variables and helper methods in your replies.

Delays

Delays are a common pattern of chatbot design. After a block of text, it’s recommended to pause for a bit to give the user a chance to read the message. The duration of the delay depends on the length of the message sent.

Stealth will pause for the duration specified. For service integrations that support it (like Facebook), Stealth will send a typing indicator while it is paused.

Naming Conventions

Replies are named after a flow’s state (which is also the controller’s action). So for a given controller:

class QuotesController < BotController
  def say_price

  end
end

You would need to place your reply template in replies/quotes/say_price.yml. If your template contains ERB, you must add the .erb suffix to the template filename: replies/quotes/say_price.yml.erb.

CatchAll

Stealth CatchAlls are designed to handle a very common scenario within chatbots. What happens when the user says something the bot doesn’t understand? The majority of bots will simply respond back with a generic “I don’t understand” and hope the user to figures out the next step. While this experience might be ok for some bots, we built a more robust way of handling these experiences right into Stealth. The better your CatchAlls, the better your bot.

Triggering

A CatchAll flow is automatically triggered when a controller action fails to do at least one of the following:

  1. Update a session (via step_to, update_session_to, or any other of the step methods)
  2. Send a reply via send_replies

In addition to the above two conditions, if your controller action raises an Exception, the CatchAll flow will automatically be triggered.

Multi-Level

Stealth keeps track of how many times a CatchAll is triggered for a given session. This allows you to build experiences in which the user is provided different responses for subsequent failures. Once a session progresses past a failing state, the CatchAll counter resets.

Retrying

By default, a Stealth bot comes with the first level CatchAll already defined. Here is the CatchAllsController action and associated reply:

def level1
  send_replies

  if previous_session_specifies_fails_to?
    step_to flow: previous_session.flow_string, state: previous_state.to_s
  else
    step_to session: previous_session - 2.states
  end
end
- reply_type: text
  text: Oops. It looks like something went wrong. Let's try that again

In the controller action, we check if the previous_session (the one that failed) specified a fails_to state. If so, we send the user there. Otherwise, we send the user back 2 states.

Sending a user back two states is a pretty good generic action. Going back 1 state takes us back to the action that failed. Since the actions most likely to fail are get actions, or actions that deal with user responses, going back 2 states usually takes us back to the original “question”.

Adding More Levels

If you would like to expand the experience, simply add a level2 controller action and associated reply (and update the FlowMap). You can go as far as you want. CatchAlls have no limit, just make sure you increment using the standardized method names of level1, level2, level3, level4, etc.

If a user has encountered the maximum number of CatchAll levels as you have defined, the user’s session will remain at the last CatchAll state.

Messaging Integrations

Stealth is designed for your bot to support one or more messaging integrations. For example, this could be just SMS or both SMS and Facebook Messenger. Messaging integrations can be attached to your Stealth bot by adding the messaging integration gem to your Gemfile.

Gemfile

source 'https://rubygems.org'

ruby '2.5.1'

gem 'stealth', '~> 0.10.0'

# Uncomment to enable the Stealth Facebook Driver
# gem 'stealth-facebook'

# Uncomment to enable the Stealth Twilio SMS Driver
# gem 'stealth-twilio'

services.yml

default: &default
  # ==========================================
  # ===== Example Facebook Service Setup =====
  # ==========================================
  # facebook:
  #   verify_token: XXXFACEBOOK_VERIFY_TOKENXXX
  #   page_access_token: XXXFACEBOOK_ACCESS_TOKENXXX
  #   setup:
  #     greeting: # Greetings are broken up by locale
  #       - locale: default
  #         text: "Welcome to the Stealth bot 🤖"
  #     persistent_menu:
  #       - type: payload
  #         text: Main Menu
  #         payload: main_menu
  #       - type: url
  #         text: Visit our website
  #         url: https://example.com
  #       - type: call
  #         text: Call us
  #         payload: "+4155330000"
  #
  # ===========================================
  # ======== Example SMS Service Setup ========
  # ===========================================
  # twilio:
  #   account_sid: XXXTWILIO_ACCOUNT_SIDXXX
  #   auth_token: XXXTWILIO_AUTH_TOKENXXX
  #   from_phone: +14155330000

production:
  <<: *default
development:
  <<: *default
test:
  <<: *default

stealth setup

Most messaging integrations require an initial setup. For example, Facebook requires you to send a payload to define the default greeting and persistent menu. You can accomplish this by running the stealth setup followed by the integration. For example:

stealth setup facebook

Make sure to reference the respective messaging integration documentation for more specifics.

Officially Supported

While we plan to add more integrations in the future, please feel free to add your own and let us know so we can keep this list updated. 😎

NLP Integrations

Stealth can be extended with NLP/NLU integrations. While these are not needed for the majority of interactions within chatbot, you may find they are helpful for certain types of interactions.

Officially Supported

Stealth currently supports:

While we plan to add more integrations in the future, please feel free to add your own and let us know so we can keep this list updated. 😎

Analytics Integrations

Stealth can be extended with metric/analytic integrations.

Officially Supported

Stealth currently supports:

While we plan to add more integrations in the future, please feel free to add your own and let us know so we can keep this list updated. 😎

Commands

Stealth provides the stealth command-line program. It is used to generate new bots, generate flows, start a console, run integration setup tasks, run the server, and more.

To view details for a command at any time use stealth help.

Usage:

  stealth [<flags>] <command> [<args> ...]

Flags:

  -h, --help           Output usage information.
  -C, --chdir="."      Change working directory.
  -v, --verbose        Enable verbose log output.
      --format="text"  Output formatter.
      --version        Show application version.

Commands:
stealth console             # Starts a stealth console
stealth db:create           # Creates the database from DATABASE_URL or config/database.yml for the current STEALTH_ENV
stealth db:create:all       # Creates all databases from DATABASE_URL or config/database.yml
stealth db:drop             # Drops the database from DATABASE_URL or config/database.yml for the current STEALTH_ENV
stealth db:drop:all         # Drops all databases from DATABASE_URL or config/database.yml
stealth db:environment:set  # Set the environment value for the database
stealth db:migrate          # Migrate the database
stealth db:rollback         # Rolls the schema back to the previous version
stealth db:schema:dump      # Creates a db/schema.rb file that is portable against any DB supported by Active Record
stealth db:schema:load      # Loads a schema.rb file into the database
stealth db:seed             # Seeds the database with data from db/seeds.rb
stealth db:setup            # Creates the database, loads the schema, and initializes with the seed data (use db:reset to also drop the database first)
stealth db:structure:dump   # Dumps the database structure to db/structure.sql. Specify another file with SCHEMA=db/my_structure.sql
stealth db:structure:load   # Recreates the databases from the structure.sql file
stealth db:version          # Retrieves the current schema version number
stealth generate            # Generates scaffold Stealth files
stealth help [COMMAND]      # Describe available commands or one specific command
stealth new                 # Creates a new Stealth bot
stealth server              # Starts a stealth server
stealth sessions:clear      # Clears all sessions in development
stealth setup               # Runs setup tasks for a specified service
stealth version             # Prints stealth version

Examples:

  Start a new Stealth project.
  $ steath new [BOT NAME]

  Generate a new flow inside your Stealth project.
  $ stealth generate flow [FLOW NAME]

  Run setup tasks for a specific driver.
  $ stealth setup [INTEGRATION NAME]

  Start a Stealth console.
  $ stealth c

  Start the Stealth server.
  $ stealth s

Deployment

Stealth is a rack based application. That means it can be hosted on most platforms as well as taking advantage of existing tools such as Docker.

Deploying on Heroku

Stealth supports Heroku out of the box. In fact, running a stealth s command locally boots foreman using a Procfile.dev file similar to what Heroku does. Here is a quick guide to get you started.

If you haven’t, make sure to track your bot in Git

  $ git init
  Initialized empty Git repository in .git/
  $ git add .
  $ git commit -m "My first commit"
  Created initial commit 5df2d09: My first commit
   42 files changed, 470 insertions(+)
    create mode 100644 Gemfile
    create mode 100644 Gemfile.lock
    create mode 100644 Procfile
  ...

After you have your bot tracked with Git, you’re ready to deploy to Heroku. Next, we’ll add our bot to Heroku using:

$ heroku apps:create <BOT NAME>

You will want a production Procfile separate from your development Procfile.dev. We recommend adding:

web: bundle exec puma -C config/puma.rb
sidekiq: bundle exec sidekiq -C config/sidekiq.yml -q webhooks -q default -r ./config/boot.rb
release: bundle exec rake db:migrate

Then deploy your bot to Heroku.

$ git push heroku master

Once deployed:

  1. Make sure to enable both the Heroku Postgres (if you use a database) and Heroku Redis addons
  2. Make sure the web and sidekiq dynos are spun up
  3. Make sure you run any stealth setup commands to configure your messaging service