Skip to content

Commit

Permalink
fix: typos
Browse files Browse the repository at this point in the history
  • Loading branch information
0xSage committed Oct 10, 2023
1 parent bac2257 commit 7be8cb5
Showing 1 changed file with 10 additions and 4 deletions.
14 changes: 10 additions & 4 deletions docs/docs/hardware/overview/cloud-vs-self-hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Cloud vs. Self-hosting Your AI
---

The choice of where to run your AI - on GPU cloud services, on-prem, vs. outright subscribing to an API provider - involves various trade-offs. The following is a naive exploration of the pros and cons of renting vs self-hosting.
The choice of how to run your AI - on GPU cloud services, on-prem, or just using an API provider - involves various trade-offs. The following is a naive exploration of the pros and cons of renting vs self-hosting.

## Cost Comparison

Expand All @@ -15,29 +15,35 @@ The following estimations use these general assumptions:
- 800 average tokens (input & output) in a single request
- Inference speed is at 24 tokens per second

### Low Usage

When operating at low capacity:

| | Self-Hosted | GPT 4.0 | GPU Rental |
| ---------------- | ----------- | ------- | ---------- |
| Cost per Request | $2.33 | $0.10 | $0.04 |

### High Usage

When operating at high capacity, i.e. 24 hours in a day, ~77.8k requests per month:

| | Self-Hosted | GPT 4.0 | GPU Rental |
| -------------- | ------------ | ------- | ---------- |
| Cost per Month | $166 (fixed) | $7465 | $3182 |

The incremental cost for large context use cases. For example, if you had to write a 500 word essay summarizing Tolstoy's "War and Peace":
### Incremental Costs

Large context use cases are also interesting to evaluate. For example, if you had to write a 500 word essay summarizing Tolstoy's "War and Peace":

| | Self-Hosted | GPT 4.0 | GPU Rental |
| ----------------------- | -------------------- | ------- | ---------- |
| Cost of "War and Peace" | (upfront fixed cost) | $94 | $40 |

> **Takeaway**: Renting on cloud or using an API is great for initially scaling. However, it can quickly become expensive when dealing with large datasets and context windows. For predictable costs, self-hosting is an attractive option.
## Business Comparison
## Business Considerations

Other considerations include
Other business level considerations may include:

| | Self-Hosted | GPT 4.0 | GPU Rental |
| ----------------------- | ----------- | ------- | ---------- |
Expand Down

0 comments on commit 7be8cb5

Please sign in to comment.