Building Your Own Independent SMS Gateway

[powerranger29]

Or at least tunnel the localhosted server somehow? I use 4g sim card internet with dynamic IP address and cannot access my local server from outside. Any solutions?

[powerranger29]

@capcom6 Also, if nothing else, would adding static ip with mobile internet provider solve that issue with local server?

[capcom6]

Hello!

Currently, the project doesn't natively support private servers. For now, you can use the provided public cloud server sms.capcom.me as outlined in our Getting Started guide.

If you're looking to connect to a local server over a connection with a dynamic IP, one workaround might be to set up a VPN from your device to a server that has a static public IP, along with the necessary routing configurations. However, instructions for setting up such a system are beyond the scope of this project.

The more complex route involves running your own server instance on a VPS (Virtual Private Server) or a similar service with a public IP. While the server component of the project isn't open-source-ready, there is a Docker image available, albeit without documentation at the moment. You'd also need to rebuild the Android app with your server address and updated Firebase credentials. If you're leaning towards this option, I will need some time to prepare and provide you with the necessary documentation for the server part.

Regarding your comment about a static IP: yes, having a static IP with your mobile internet provider could potentially resolve the issue with accessing your local server externally. However, that approach may have additional cost implications and would still require some network configuration.

I hope this clarifies your options. Let me know which path seems most suitable for you, and I'll do my best to assist you further.

[powerranger29]

for starters, id like to go with a dynamic ip (i have an option with my provider) Can you please guide me what network configuration should be done here like you mentioned?

I tried with the VPN before but could not manage it out of the box. VPN was active on my phone (through vpn app, not android vpn settings) but when trying to run sms gateway app locally it crashed (like it does when trying to run local server through my mobile internet). Any suggestion here maybe?

In the future it would be awesome if you could also open source the server component / add option to add custom server inside app.

[capcom6]

Actually, if you're considering using a dynamic IP, the most straightforward solution remains the public cloud server I previously mentioned. However, if you're referring to a static IP in your first sentence, then you should be able to see an external address for connection in the Local server section on the main app screen. You can try connecting to this address from another device. If that doesn't work, it would be a good idea to check with your provider to see which ports they allow for incoming connections to your IP. But see the last paragraph about crash.

Regarding the VPN issue, to successfully connect through a VPN, you'd typically need control over a server where you could adjust routing settings. Some VPN apps and services might offer such functionality, but I'm not aware of any specific ones to recommend at this time.

As for the app crash, I've managed to reproduce the issue and am planning to address it in an upcoming update. I will keep you posted and provide feedback as soon as the new version is available.

Thanks for your report.

[capcom6]

Hello!

I've released an update that should fix the crash on startup. Please download and try out version v1.2.3 and let me know how it goes.

[hacen-ai]

@capcom6 +1 for private cloud, I have my own vps and it will be realy cool if you can provide any documentation on how to edit the app so I can use it with my own cloud.BTW great work here thank you for make in it public for us, many thanks

[capcom6]

Hello!

Thanks for your feedback. I've started writing instructions for running your own private server for the app, but I found that the current version of the server is not quite suitable for private use.

Firstly, there are some not-so-clear steps:

1. Registration in the Firebase console and setting up Cloud Messaging on both the backend and the mobile app.
2. Rebuilding the Android app with source code changes.

And second, but more importantly, the current version of the backend is designed for public use. There are some publicly available endpoints that are inappropriate for private use. For example, anyone who knows your server address can register the app on the server.

While I will still publish the high-level instruction in a few days, please note that running this version of the server is not recommended for private use due to these challenges.

However, for the future, I plan to add one more mode to the application, in addition to "Local" and "Cloud" - "Private". For this mode, I'll be developing a simple server API-compatible with the Cloud mode, but for Push notifications, the Private server will leverage the Cloud server. Push notifications won't contain any information about the message (text or phones) for now, minimizing privacy impact. As for the mobile app, you'll set your own private server address in Private mode along with some credentials, so the mobile app will communicate only with your private server.

With this architecture, there's no need to rebuild the mobile app or register your own project in Firebase. Just start a minimal backend on your side and activate Private mode in the mobile app.

I'd appreciate any opinions about my architectural idea.

PS: I'll be releasing end-to-end encryption support for the app soon. This feature will allow you to encrypt messages before sending them via API, and they will only be decrypted in the mobile app. So if you're concerned about accessing message text or phone numbers on the backend side, this feature will address those concerns.

Thanks!

[capcom6]

Here is the quick start guide for deploying your own backend for the app.

Before You Start

At the time of writing this document, the backend is not optimized for use as a private server. It contains public endpoints, and therefore, anyone who knows the server address can connect a mobile app to it and use the API.

Please note that this is a simplified instruction that does not provide detailed security settings, such as firewall configuration and HTTPS setup. It is the user's responsibility to implement these security measures.

Use at your own risk.

Prerequisites

1. Android Studio with the Android SDK.
2. A Google Firebase account and a project with active Cloud Messaging.
3. A MySQL or MariaDB database.
4. A VPS with a public IP address.
5. A reverse proxy like Nginx or Traefik with HTTPS support.

Server

1. Generate a Firebase service account private key.
2. Create a config file in YAML. Some variables can be overridden with environment variables. Example config.yml:

http:
  listen: ":3000"   # HTTP__LISTEN
database:
  dialect: "mysql"  # DATABASE__DIALECT
  host: "localhost" # DATABASE__HOST
  port: 3306        # DATABASE__PORT
  user: "sms"       # DATABASE__USER
  password: "sms"   # DATABASE__PASSWORD
  database: "sms"   # DATABASE__DATABASE
  timezone: "UTC"   # DATABASE__TIMEZONE
fcm:
  credentials_json:  >
    {
    ...
    }
tasks:
  hashing:
    interval_seconds: 900

3. Create a docker-compose.yaml file like this:

version: '3'
services:
  backend:
    image: capcom6/sms-gateway
    environment:
      - CONFIG_PATH=config.yml
      - HTTP__LISTEN=0.0.0.0:3000
      - DATABASE__HOST=db
      - DATABASE__PORT=3306
      - DATABASE__USER=sms
      - DATABASE__PASSWORD=sms
      - DATABASE__DATABASE=sms
    ports:
      - "3000:3000"
    volumes:
      - ./config.yml:/app/config.yml:ro
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy

  db:
    image: mariadb:10.11
    environment:
      - MYSQL_ROOT_PASSWORD=root
      - MYSQL_DATABASE=sms
      - MYSQL_USER=sms
      - MYSQL_PASSWORD=sms
    volumes:
      - mariadb-data:/var/lib/mysql
    restart: unless-stopped
    healthcheck:
      test:
        [
          "CMD",
          "mysqladmin",
          "ping",
          "-proot",
          "-h",
          "127.0.0.1"
        ]
      timeout: 5s
      retries: 10

volumes:
  mariadb-data:

4. Run docker-compose up -d.
5. Configure the reverse proxy to pass traffic to your server.

Client

To build a version of the Android application that will communicate with your own server, follow these steps:

1. Clone the repository and open it in Android Studio.
2. Open app/src/main/java/me/capcom/smsgateway/modules/gateway/GatewayApi.kt and change the BASE_URL constant value to your server's URL.
3. Change the applicationId in app/build.gradle to a new value.
4. Follow the instructions from https://firebase.google.com/docs/android/setup to register your application and generate the google-services.json file.
5. Build and run the application.

[capcom6]

I apologize, but I can only assist you with the following modes:

1. Local mode
2. Public Cloud Server mode
3. Private Server mode

For building your own gateway, you'll need to take on that responsibility. I recommend referring to the documentation I mentioned earlier, particularly the section on Custom Gateway Setup. Please note that modifying the core architecture (like use of FCM from Private Server directly) is not covered in the documentation and would require independent development on your part.

[rohit9625]

No, problem. This documentation only says that we need to build binaries or docker images for server. What about the config files? Do we still need to add config.yml and docker-compose.yml?

[capcom6]

Yes, you'll need to configure your server. An example config.yml can be found at: config.example.yml Also, you can use environment variables.

Using docker-compose.yml is optional but can simplify deployment. If you prefer, you can run the Docker container directly without Docker Compose. Or use any orchestrator of your choice.

[rohit9625]

Hey @capcom6, thanks for your help. Have you ever noticed that the onReceive() method in MessagesReceiver is not triggered when messages are received from other phone that has Google's Messenger App? However, it do triggers when I send the message to my number(my phone has Samsung's Default Messenger App) or I receive some promotional messages from a third party.

What can we do in the android-sms-gateway app to handle this?

[capcom6]

Please refer to the FAQ in part of RCS: https://docs.sms-gate.app/faq/webhooks/#the-smsreceived-webhook-is-not-triggering

[williamdes]

With this architecture, there's no need to rebuild the mobile app or register your own project in Firebase. Just start a minimal backend on your side and activate Private mode in the mobile app.
I'd appreciate any opinions about my architectural idea.

That would be awesome, I am looking into this option for sure !

You can use the concept of a pre-shared secret between the back end (self hosted) and the clients.
That will make attempts of hacking quite difficult.
Or use SSL key pairs

[ApuRed]

Hello, your manual is very good, I managed to install the server and the Android application, but I have a problem, the SMS arrive after 15, 20 or 30 minutes after being sent, I am not an Android expert, could you help me remove the delay?

[capcom6]

Hello, @ApuRed

Thank you for reaching out.

Regarding the delay in SMS delivery, it does seem to be an issue related to Firebase Cloud Messaging (FCM). Here are a few things you can check to troubleshoot this issue:

1. Server Logs: Look for any errors in the server's log regarding sending notifications. This could provide clues if the server is encountering issues when attempting to send out push notifications.
2. Firebase Credentials: Ensure that the Firebase credentials are correctly set in the config.yml file. Incorrect credentials can lead to failed notification delivery.
3. Google Play Services: Verify that Google Play Services or an open-source alternative is installed and updated on the Android device. FCM relies on Google Play Services for notification delivery.

Additionally, please check the devices table in the database to see if the push token is present and correctly filled in for the device in question.

I also wanted to share some good news: I plan to open source and release a server version with private mode support in a few weeks. With this upcoming release, there will be no need for you to rebuild the client and set up FCM on your own.

I hope this helps, and please let me know if you have any further questions or issues.

[IOAPP-IO]

Hello, @ApuRed

Thank you for reaching out.

Regarding the delay in SMS delivery, it does seem to be an issue related to Firebase Cloud Messaging (FCM). Here are a few things you can check to troubleshoot this issue:

1. Server Logs: Look for any errors in the server's log regarding sending notifications. This could provide clues if the server is encountering issues when attempting to send out push notifications.
2. Firebase Credentials: Ensure that the Firebase credentials are correctly set in the config.yml file. Incorrect credentials can lead to failed notification delivery.
3. Google Play Services: Verify that Google Play Services or an open-source alternative is installed and updated on the Android device. FCM relies on Google Play Services for notification delivery.

Additionally, please check the devices table in the database to see if the push token is present and correctly filled in for the device in question.

I also wanted to share some good news: I plan to open source and release a server version with private mode support in a few weeks. With this upcoming release, there will be no need for you to rebuild the client and set up FCM on your own.

I hope this helps, and please let me know if you have any further questions or issues.

You are an amazing person, thank you very much

[IOAPP-IO]

is the documentation ready or still?

[capcom6]

@IOAPP-IO , hello.

I'm currently finishing up the first version of the documentation for setting up a private server. I expect this to be completed by the end of next week, after which I will post updated documentation to make working in private mode easier.

[capcom6]

Hello everyone!

I'm excited to share that I've made significant progress with the app. The server component has now been open-sourced, and I've put together a quick start guide to facilitate running your app server in private mode. Please note that to use the private mode, you'll need version 1.10.0 or higher of the Android app.

You can find the documentation for setting up a private server here: https://sms.capcom.me/getting-started/private-server/

The source code is also now available in the GitHub repository at: https://github.com/capcom6/sms-gateway

Keep in mind that the private mode feature is still in the testing phase and may need further improvements. Your feedback during this phase would be incredibly valuable.

Thank you all for your interest and patience. I'm marking this issue as closed, but if you encounter any problems or have any suggestions, please don't hesitate to open a new issue either here or in the server's repository, depending on where the issue lies.

[PointerSal]

Hello,

first of all, great work here. I would like to use my firebase account (setup) and my own VPS. Steps mentioned on January 26 are still valid?

I see that on the "Getting Started - Private Server" you're still mentioning the use of the public server and the native firebase account. With the steps mentioned on the 26, would it be still necessary?

[capcom6]

Hello, @PointerSal

Yes, the steps outlined on January 26 for setting up your own independent server are still applicable with a few minor modifications:

1. By default, the server operates in public mode. For more details, please see the "Before You Start" section.
2. The server's Private Mode now uses a predetermined upstream URL. If you plan to utilize Private Mode as well, you can modify this at internal/sms-gateway/modules/push/upstream/client.go to pointing to your main server.
3. It's no longer necessary to edit GatewayApi.kt within the Android app since you can now update the server URL from the app's settings. However, changing the URL in the file can still be useful for setting a default value.

Note that the Private Server mode is designed to streamline the process of setting up your server without the need for rebuilding anything. It is a simplified approach as compared to the fully independent server setup described on January 26.

[PointerSal]

Thanks @capcom6 for your answer.

If I understood correctly, with such a setup the private server will still query the public one for sending FCM notifications.

I'd like to use my Firebase account: any change I need to do in the server code for pointing directly to my firebase project APIs?

[capcom6]

@PointerSal

To clarify, there's a distinction between the Private Server mode, which requires minimal setup, and your own custom-built gateway. The steps provided on January 26, along with the updates on May 27, guide you through the process of setting up an independent public server and corresponding Android app using your Firebase account, thus eliminating any dependencies on my server at sms.capcom.me.

In these steps, you would replace references to sms.capcom.me with the URL of your own public server, among other configurations. But please be aware, without additional modifications to the source code or setting up a reverse proxy, the server will operate in public mode by default. This means that anyone with your server's address could potentially register devices.