Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

## Bug Fixes

- fix: `:compression_level` is reachable. It was unusable on every path — no adapter declares it, so `NimbleOptions` rejected it on compile-time adapter opts, and it resolved to `nil` before it could reach the encoder otherwise. It is now an option on the `use Cache` line (`compression_level: 6`), it is taken off the adapter opts before they are validated so the `opts: [compression_level: 6]` spelling works too, and it is never handed to the adapter. Setting it forces encoding on adapters that hold terms natively — asking for compression is asking for bytes. A cache using a strategy adapter raises at compile time rather than ignoring the option.
- fix: `Cache.ConCache.get_or_store/3` followed by `get/1` no longer raises. `get_or_store/3` writes through ConCache directly, bypassing the encode in `put/3`, so the matching `get/1` tried to `binary_to_term/1` a raw term.
- fix: a binary value wrapped in braces but not valid JSON (eg `"{oops}"`) no longer raises `Jason.DecodeError` on read. `Cache.TermEncoder.decode/1` used `Jason.decode!/1`, and now falls back to returning the binary unchanged.
- fix: raw `Cache.ETS` operations (`match_object/1`, `select/1`, `tab2list/0`, `foldl/2`) now see the terms that were `put`, rather than the opaque encoded binaries they used to return.
Expand Down
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,27 @@ MyModule.put("key", "value") #> :ok
MyModule.get("key") #> {:ok, "value"}
```

### Compression

Values are serialised with `:erlang.term_to_binary/1` for adapters that cannot hold an
Erlang term. Pass a `compression_level` (`1..9`) to compress them on the way in:

```elixir
defmodule MyModule do
use Cache,
adapter: Cache.Redis,
name: :my_name,
compression_level: 6,
opts: [...opts]
end
```

The level belongs to the encoder rather than to the store, so it is never handed to the
adapter. Setting it also forces encoding on adapters that would otherwise hold the term
as it is (`Cache.ETS`, `Cache.Agent`, `Cache.PersistentTerm`, `Cache.ConCache`) — asking
for compression is asking for bytes. It is not supported on a cache using a strategy
adapter, which encodes through what it wraps; set it there instead.

## Adapters
- `Cache.Agent` - Simple agent based caching
- `Cache.DETS` - Disk persisted caching with [dets](https://www.erlang.org/doc/man/dets.html)
Expand Down
39 changes: 35 additions & 4 deletions lib/cache.ex
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,21 @@ defmodule Cache do
adapter_opts = check_adapter_opts.(unquote(opts[:opts]))

@adapter_opts adapter_opts
@compression_level if is_list(@adapter_opts), do: @adapter_opts[:compression_level]

# A strategy encodes inside the strategy module, through the adapter or the cache
# modules it wraps, so a `:compression_level` set here would never reach an
# encoder. Say that at compile time rather than quietly handing back uncompressed
# values.
if not is_nil(unquote(opts[:compression_level])) or
(is_list(@adapter_opts) and not is_nil(@adapter_opts[:compression_level])) do
raise ArgumentError, """
`:compression_level` is not supported on a cache using a strategy adapter.

Got: #{inspect(@cache_strategy_module)} in #{inspect(__MODULE__)}

Set it on the cache module or layer that does the storing instead.
"""
end

def cache_name, do: @cache_name
def cache_adapter, do: @cache_adapter
Expand Down Expand Up @@ -299,16 +313,33 @@ defmodule Cache do
pre_check_runtime_options.(adapter_opts)
end

adapter_opts = if opts[:sandbox?], do: [], else: check_adapter_opts.(opts[:opts])
# `:compression_level` belongs to the encoder, not to the store, so it comes off
# the adapter opts before they are validated. No adapter declares it, and an
# adapter that forwards its opts to the store would choke on an option that was
# never meant for it.
{configured_compression_level, configured_adapter_opts} =
case opts[:opts] do
configured_adapter_opts when is_list(configured_adapter_opts) ->
Keyword.pop(configured_adapter_opts, :compression_level)

configured_adapter_opts ->
{nil, configured_adapter_opts}
end

adapter_opts = if opts[:sandbox?], do: [], else: check_adapter_opts.(configured_adapter_opts)

@adapter_opts adapter_opts
@compression_level if is_list(@adapter_opts), do: @adapter_opts[:compression_level]
@compression_level opts[:compression_level] || configured_compression_level

# Resolved against the *configured* adapter and its *configured* opts rather than
# `@cache_adapter`/`@adapter_opts`, which are swapped out under `sandbox?: true`.
# A sandboxed cache must encode exactly like the adapter it stands in for, or a
# value would round-trip differently in test than in production.
@cache_encode_terms? Cache.TermEncoder.encoding_required?(opts[:adapter], opts[:opts])
#
# A compression level is an explicit request to compress, so it encodes even on an
# adapter that could have held the term as it is.
@cache_encode_terms? not is_nil(@compression_level) or
Cache.TermEncoder.encoding_required?(opts[:adapter], opts[:opts])

unquote(adapter_use_ast)

Expand Down
88 changes: 88 additions & 0 deletions test/cache/compression_level_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
defmodule Cache.CompressionLevelTest do
@moduledoc """
`:compression_level` was unreachable on every path: no adapter declares it, so
`NimbleOptions` rejected it on compile-time opts, and it never reached the encoder
on any other. These tests hold it reachable.
"""

use ExUnit.Case, async: true

@value String.duplicate("compress me ", 500)

defmodule CompressedCache do
use Cache, adapter: Cache.ETS, name: :compression_use_option, compression_level: 6, opts: []
end

defmodule AdapterOptsCache do
use Cache, adapter: Cache.ETS, name: :compression_adapter_opts, opts: [compression_level: 6]
end

defmodule RedisCompressedCache do
use Cache,
adapter: Cache.Redis,
name: :compression_redis,
compression_level: 6,
opts: [uri: "redis://localhost:6379"]
end

defmodule UncompressedCache do
use Cache, adapter: Cache.ETS, name: :compression_none, opts: []
end

setup do
start_supervised!(
{Cache, [CompressedCache, AdapterOptsCache, RedisCompressedCache, UncompressedCache]}
)

:ok
end

describe "&put/3 with a compression level" do
test "compresses the stored value and reads it back unchanged" do
assert :ok === CompressedCache.put(:compressed, @value)
assert [{:compressed, stored}] = :ets.lookup(:compression_use_option, :compressed)

assert is_binary(stored)
assert byte_size(stored) < byte_size(:erlang.term_to_binary(@value))
assert {:ok, @value} === CompressedCache.get(:compressed)
end

test "reaches the encoder from the adapter opts as well" do
assert :ok === AdapterOptsCache.put(:compressed, @value)
assert [{:compressed, stored}] = :ets.lookup(:compression_adapter_opts, :compressed)

assert byte_size(stored) < byte_size(:erlang.term_to_binary(@value))
assert {:ok, @value} === AdapterOptsCache.get(:compressed)
end

test "compresses on an adapter that stores bytes" do
assert :ok === RedisCompressedCache.put("compressed", @value)
stored = Cache.Redis.command!(:compression_redis, ["GET", "compression_redis:compressed"])

assert byte_size(stored) < byte_size(:erlang.term_to_binary(@value))
assert {:ok, @value} === RedisCompressedCache.get("compressed")
end

test "an adapter that holds terms stores the term itself without one" do
assert :ok === UncompressedCache.put(:plain, @value)
assert [{:plain, stored}] = :ets.lookup(:compression_none, :plain)

assert stored === @value
assert {:ok, @value} === UncompressedCache.get(:plain)
end
end

describe "&__using__/1 with a strategy adapter" do
test "refuses a compression level rather than ignoring it" do
assert_raise ArgumentError, ~r/not supported on a cache using a strategy adapter/, fn ->
defmodule StrategyCompressedCache do
use Cache,
adapter: {Cache.HashRing, Cache.ETS},
name: :compression_strategy,
compression_level: 6,
opts: []
end
end
end
end
end
Loading