Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,15 @@ To run a backup, launch `mysql-backup` - as a container or as a binary - with th
For example:

````bash
docker run -d --restart=always -e DB_DUMP_FREQUENCY=60 -e DB_DUMP_BEGIN=2330 -e DB_DUMP_TARGET=/local/file/path -e DB_SERVER=my-db-address -v /local/file/path:/db databack/mysql-backup dump
docker run -d --restart=always -e DB_DUMP_FREQUENCY=60 -e DB_DUMP_BEGIN=2330Z -e DB_DUMP_TARGET=/local/file/path -e DB_SERVER=my-db-address -v /local/file/path:/db databack/mysql-backup dump

# or

mysql-backup dump --frequency=60 --begin=2330 --target=/local/file/path --server=my-db-address
mysql-backup dump --frequency=60 --begin=2330Z --target=/local/file/path --server=my-db-address

# or to connect to a local mysqld via the unix domain socket as the current user

mysql-backup dump --frequency=60 --begin=2330 --target=/local/file/path --server=/run/mysqld/mysqld.sock
mysql-backup dump --frequency=60 --begin=2330Z --target=/local/file/path --server=/run/mysqld/mysqld.sock
````

Or `mysql-backup --config-file=/path/to/config/file.yaml` where `/path/to/config/file.yaml` is a file
Expand All @@ -71,14 +71,14 @@ dump:
target: /local/file/path
```

The above will run a dump every 60 minutes, beginning at the next 2330 local time, from the database accessible in the container `my-db-address`.
The command and environment-variable examples run a dump every 60 minutes, beginning at the next 23:30 local time. The config-file example uses the legacy zoneless form, which is interpreted as UTC.

````bash
docker run -d --restart=always -e DB_USER=user123 -e DB_PASS=pass123 -e DB_DUMP_FREQUENCY=60 -e DB_DUMP_BEGIN=2330 -e DB_DUMP_TARGET=/db -e DB_SERVER=my-db-address -v /local/file/path:/db databack/mysql-backup dump
docker run -d --restart=always -e DB_USER=user123 -e DB_PASS=pass123 -e DB_DUMP_FREQUENCY=60 -e DB_DUMP_BEGIN=2330Z -e DB_DUMP_TARGET=/db -e DB_SERVER=my-db-address -v /local/file/path:/db databack/mysql-backup dump

# or

mysql-backup dump --user=user123 --pass=pass123 --frequency=60 --begin=2330 --target=/local/file/path --server=my-db-address --port=3306
mysql-backup dump --user=user123 --pass=pass123 --frequency=60 --begin=2330Z --target=/local/file/path --server=my-db-address --port=3306
````

See [backup](./docs/backup.md) for a more detailed description of performing backups.
Expand Down
92 changes: 78 additions & 14 deletions cmd/dump.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"fmt"
"os"
"strings"
"time"
_ "time/tzdata"

"github.com/google/uuid"
"github.com/spf13/cobra"
Expand Down Expand Up @@ -240,7 +242,10 @@ func dumpCmd(passedExecs execs, cmdConfig *cmdConfiguration) (*cobra.Command, er
}

// timer options
timerOpts := parseTimerOptions(v, cmdConfig.configuration)
timerOpts, err := parseTimerOptions(v, cmdConfig.configuration)
if err != nil {
return err
}

var executor execs
executor = &core.Executor{}
Expand Down Expand Up @@ -387,17 +392,7 @@ S3: If it is a URL of the format s3://bucketname/path then it will connect via S
// skip extended insert in dump; instead, one INSERT per record in each table
flags.Bool("skip-extended-insert", false, "Skip extended insert in dump; instead, one INSERT per record in each table.")

// frequency
flags.Int("frequency", defaultFrequency, "how often to run backups, in minutes")

// begin
flags.String("begin", defaultBegin, "What time to do the first dump. Must be in one of two formats: Absolute: HHMM, e.g. `2330` or `0415`; or Relative: +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half")

// cron
flags.String("cron", "", "Set the dump schedule using standard [crontab syntax](https://en.wikipedia.org/wiki/Cron), a single line.")

// once
flags.Bool("once", false, "Override all other settings and run the dump once immediately and exit. Useful if you use an external scheduler (e.g. as part of an orchestration solution like Cattle or Docker Swarm or [kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)) and don't want the container to do the scheduling internally.")
addTimerFlags(flags)

// parallelism - how many databases (and therefore connections) to back up at once
flags.Int("parallelism", 1, "How many databases to back up in parallel.")
Expand Down Expand Up @@ -444,7 +439,7 @@ S3: If it is a URL of the format s3://bucketname/path then it will connect via S
return cmd, nil
}

func parseTimerOptions(v *viper.Viper, config *api.ConfigSpec) core.TimerOptions {
func parseTimerOptions(v *viper.Viper, config *api.ConfigSpec) (core.TimerOptions, error) {
var scheduleConfig *api.Schedule
if config != nil {
dumpConfig := config.Dump
Expand All @@ -464,6 +459,75 @@ func parseTimerOptions(v *viper.Viper, config *api.ConfigSpec) core.TimerOptions
if begin == "" && scheduleConfig != nil && scheduleConfig.Begin != nil {
begin = fmt.Sprintf("%d", *scheduleConfig.Begin)
}
if begin != "" && !strings.HasPrefix(begin, "+") {
var parsed time.Time
var err error
clock, zoneName, hasZoneName := strings.Cut(begin, "@")
switch {
case hasZoneName:
parsed, err = func() (time.Time, error) {
clockTime, err := time.Parse("1504", clock)
if err != nil {
return time.Time{}, err
}

var location *time.Location
switch zoneName {
case "local":
location = time.Local
default:
location, err = time.LoadLocation(zoneName)
if err != nil {
return time.Time{}, err
}
}

now := time.Now()
localNow := now.In(location)
requestedHour := clockTime.Hour()
requestedMinuteOfHour := clockTime.Minute()
requestedMinute := requestedHour*60 + requestedMinuteOfHour
currentMinute := localNow.Hour()*60 + localNow.Minute()

// Search actual instants rather than relying on time.Date so DST
// overlaps select the earliest future occurrence, and DST gaps
// can be detected rather than silently normalized.
findOccurrence := func(year int, month time.Month, day int) (time.Time, bool) {
anchor := time.Date(year, month, day, 12, 0, 0, 0, time.UTC)
for candidate := anchor.Add(-30 * time.Hour); !candidate.After(anchor.Add(30 * time.Hour)); candidate = candidate.Add(time.Minute) {
wall := candidate.In(location)
if wall.Year() == year && wall.Month() == month && wall.Day() == day &&
wall.Hour() == requestedHour && wall.Minute() == requestedMinuteOfHour && candidate.After(now) {
return candidate, true
}
}
return time.Time{}, false
}

if occurrence, found := findOccurrence(localNow.Year(), localNow.Month(), localNow.Day()); found {
return occurrence, nil
}
if requestedMinute > currentMinute {
return time.Time{}, fmt.Errorf("time %s does not exist today in timezone %s", clock, zoneName)
}

tomorrow := localNow.AddDate(0, 0, 1)
if occurrence, found := findOccurrence(tomorrow.Year(), tomorrow.Month(), tomorrow.Day()); found {
return occurrence, nil
}
return time.Time{}, fmt.Errorf("time %s does not exist tomorrow in timezone %s", clock, zoneName)
}()
case len(begin) == 4:
// Preserve the legacy behavior: an absolute time without a zone is UTC.
parsed, err = time.Parse("1504", begin)
default:
parsed, err = time.Parse("1504Z07:00", begin)
}
if err != nil {
return core.TimerOptions{}, fmt.Errorf("invalid begin option %q: %w", begin, err)
}
begin = parsed.UTC().Format("1504")
}
frequency := v.GetInt("frequency")
if frequency == 0 && scheduleConfig != nil && scheduleConfig.Frequency != nil {
frequency = *scheduleConfig.Frequency
Expand All @@ -473,7 +537,7 @@ func parseTimerOptions(v *viper.Viper, config *api.ConfigSpec) core.TimerOptions
Cron: cron,
Begin: begin,
Frequency: frequency,
}
}, nil

}

Expand Down
93 changes: 93 additions & 0 deletions cmd/dump_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,109 @@ import (
"io"
"net/url"
"testing"
"time"

"github.com/databacker/mysql-backup/pkg/compression"
"github.com/databacker/mysql-backup/pkg/core"
"github.com/databacker/mysql-backup/pkg/database"
"github.com/databacker/mysql-backup/pkg/storage"
"github.com/databacker/mysql-backup/pkg/storage/file"
"github.com/go-test/deep"
"github.com/spf13/viper"
"github.com/stretchr/testify/mock"
)

func TestParseTimerOptionsBegin(t *testing.T) {
t.Parallel()

tests := []struct {
name string
begin string
expected string
wantError bool
}{
{name: "relative time", begin: "+25", expected: "+25"},
{name: "legacy implicit UTC", begin: "0400", expected: "0400"},
{name: "explicit UTC", begin: "0400Z", expected: "0400"},
{name: "positive offset crossing midnight", begin: "0400+08:00", expected: "2000"},
{name: "negative offset", begin: "0400-05:30", expected: "0930"},
{name: "invalid hour", begin: "2500Z", wantError: true},
{name: "invalid offset", begin: "0400+25:00", wantError: true},
{name: "invalid local time", begin: "2500@local", wantError: true},
{name: "unknown timezone", begin: "0400@Not/A_Real_Zone", wantError: true},
{name: "invalid suffix", begin: "0400UTC", wantError: true},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
v := viper.New()
v.Set("begin", tt.begin)

options, err := parseTimerOptions(v, nil)
if tt.wantError {
if err == nil {
t.Fatalf("parseTimerOptions() error = nil, want an error")
}
return
}
if err != nil {
t.Fatalf("parseTimerOptions() error = %v", err)
}
if options.Begin != tt.expected {
t.Errorf("parseTimerOptions() Begin = %q, want %q", options.Begin, tt.expected)
}
})
}
}

func TestParseTimerOptionsLocalBegin(t *testing.T) {
t.Parallel()

now := time.Now().In(time.Local)
local := time.Date(now.Year(), now.Month(), now.Day(), 4, 0, 0, 0, time.Local)
if !local.After(now) {
local = local.AddDate(0, 0, 1)
}

v := viper.New()
v.Set("begin", "0400@local")
options, err := parseTimerOptions(v, nil)
if err != nil {
t.Fatalf("parseTimerOptions() error = %v", err)
}
if expected := local.UTC().Format("1504"); options.Begin != expected {
t.Errorf("parseTimerOptions() Begin = %q, want %q", options.Begin, expected)
}
}

func TestParseTimerOptionsNamedTimezoneBegin(t *testing.T) {
t.Parallel()

for _, zoneName := range []string{"America/New_York", "Asia/Jerusalem", "Asia/Kathmandu", "Pacific/Kiritimati"} {
t.Run(zoneName, func(t *testing.T) {
location, err := time.LoadLocation(zoneName)
if err != nil {
t.Fatalf("time.LoadLocation(%q) error = %v", zoneName, err)
}
now := time.Now().In(location)
local := time.Date(now.Year(), now.Month(), now.Day(), 4, 0, 0, 0, location)
if !local.After(now) {
local = local.AddDate(0, 0, 1)
}

v := viper.New()
v.Set("begin", "0400@"+zoneName)
options, err := parseTimerOptions(v, nil)
if err != nil {
t.Fatalf("parseTimerOptions() error = %v", err)
}
if expected := local.UTC().Format("1504"); options.Begin != expected {
t.Errorf("parseTimerOptions() Begin = %q, want %q", options.Begin, expected)
}
})
}
}

func TestDumpCmd(t *testing.T) {
t.Parallel()

Expand Down
17 changes: 17 additions & 0 deletions cmd/flags.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package cmd

import "github.com/spf13/pflag"

func addTimerFlags(flags *pflag.FlagSet) {
// frequency
flags.Int("frequency", defaultFrequency, "how often to run, in minutes")

// begin
flags.String("begin", defaultBegin, "What time to do the first run, as absolute or relative time. Absolute times may be UTC (`0400Z`), include a UTC offset (`0400+08:00`), use the platform's local timezone (`0400@local`), or use an IANA timezone (`0400@America/New_York`). Relative times use +MM, i.e. minutes after starting the run, such as `+0`, `+10`, or `+90`. A zoneless time (`0400`) is legacy and should not be used, but is interpreted as UTC.")

// cron
flags.String("cron", "", "Set the run schedule using standard [crontab syntax](https://en.wikipedia.org/wiki/Cron), a single line.")

// once
flags.Bool("once", false, "Override all other settings and run once immediately and exit. Useful if you use an external scheduler (e.g. as part of an orchestration solution like Cattle or Docker Swarm or [kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)) and don't want the container to do the scheduling internally.")
}
17 changes: 5 additions & 12 deletions cmd/prune.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,10 @@ func pruneCmd(passedExecs execs, cmdConfig *cmdConfiguration) (*cobra.Command, e
}

// timer options
timerOpts := parseTimerOptions(v, cmdConfig.configuration)
timerOpts, err := parseTimerOptions(v, cmdConfig.configuration)
if err != nil {
return err
}

var executor execs
executor = &core.Executor{}
Expand Down Expand Up @@ -92,17 +95,7 @@ func pruneCmd(passedExecs execs, cmdConfig *cmdConfiguration) (*cobra.Command, e
// retention
flags.String("retention", "", "Retention period for backups. REQUIRED. Can be number of backups or time-based. For time-based, the format is: 1d, 1w, 1m, 1y for days, weeks, months, years, respectively. For number-based, the format is: 1c, 2c, 3c, etc. for the count of backups to keep.")

// frequency
flags.Int("frequency", defaultFrequency, "how often to run prunes, in minutes")

// begin
flags.String("begin", defaultBegin, "What time to do the first prune. Must be in one of two formats: Absolute: HHMM, e.g. `2330` or `0415`; or Relative: +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half")

// cron
flags.String("cron", "", "Set the prune schedule using standard [crontab syntax](https://en.wikipedia.org/wiki/Cron), a single line.")

// once
flags.Bool("once", false, "Override all other settings and run the prune once immediately and exit. Useful if you use an external scheduler (e.g. as part of an orchestration solution like Cattle or Docker Swarm or [kubernetes cron jobs](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)) and don't want the container to do the scheduling internally.")
addTimerFlags(flags)

return cmd, nil
}
4 changes: 2 additions & 2 deletions docs/backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ mysql-backup dump --pre-backup-scripts=/path/to/pre-backup/scripts --post-backup

```bash
docker run -d --restart=always -e DB_USER=user123 -e DB_PASS=pass123 -e DB_DUMP_FREQUENCY=60 \
-e DB_DUMP_BEGIN=2330 -e DB_DUMP_TARGET=/db -e DB_SERVER=my-db-container:db \
-e DB_DUMP_BEGIN=2330Z -e DB_DUMP_TARGET=/db -e DB_SERVER=my-db-container:db \
-v /path/to/pre-backup/scripts:/scripts.d/pre-backup \
-v /path/to/post-backup/scripts:/scripts.d/post-backup \
-v /local/file/path:/db \
Expand All @@ -345,7 +345,7 @@ services:
- DB_USER=user123
- DB_PASS=pass123
- DB_DUMP_FREQUENCY=60
- DB_DUMP_BEGIN=2330
- DB_DUMP_BEGIN=2330Z
- DB_SERVER=mysql_db
command: dump
mysql_db:
Expand Down
4 changes: 2 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ The following are the environment variables, CLI flags and configuration file op
| Replace single long INSERT statement per table with one INSERT statement per line | B | `skip-extended-insert` | `DB_DUMP_SKIP_EXTENDED_INSERT` | `dump.skipExtendedInsert` | `false` |
| restore to a specific database | R | `restore --database` | `RESTORE_DATABASE` | `restore.database` | |
| how often to do a dump or prune, in minutes | BP | `dump --frequency` | `DB_DUMP_FREQUENCY` | `dump.schedule.frequency` | `1440` (in minutes), i.e. once per day |
| what time to do the first dump or prune | BP | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `0`, i.e. immediately |
| what time to do the first dump or prune; see [scheduling](./scheduling.md#frequency-and-delayed-start) | BP | `dump --begin` | `DB_DUMP_BEGIN` | `dump.schedule.begin` | `+0`, i.e. immediately |
| cron schedule for dumps or prunes | BP | `dump --cron` | `DB_DUMP_CRON` | `dump.schedule.cron` | |
| run the backup or prune a single time and exit | BP | `dump --once` | `DB_DUMP_ONCE` | `dump.schedule.once` | `false` |
| enable debug logging | BRP | `debug` | `DB_DEBUG` | `logging` | `false` |
Expand Down Expand Up @@ -135,7 +135,7 @@ for details of each.
* `noDatabaseName`: boolean, remove `USE <database>` from dumpfile
* `schedule`: the schedule configuration
* `frequency`: int, the frequency of the schedule in minutes
* `begin`: int, the time to begin the schedule in minutes from start of process
* `begin`: int, the time to begin the schedule in minutes from start of process. The CLI flag and environment variable also accept the absolute-time formats described in [scheduling](./scheduling.md#frequency-and-delayed-start).
* `cron`: string, the cron schedule
* `once`: boolean, run once and exit
* `compression`: string, the compression to use
Expand Down
21 changes: 21 additions & 0 deletions docs/scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,17 @@ The frequency value is in minutes. Thus, you can set backup to run every hour by
For a relative delayed start, prefix the number of minutes with `+`; for example, `+120` delays the
first backup by 2 hours.

An absolute delayed start uses a four-digit 24-hour time followed by its timezone:

* `0400Z` means 04:00 UTC.
* `0400+08:00` means 04:00 at a fixed UTC+08:00 offset.
* `0400@local` means 04:00 in the timezone of the computer or container. The local timezone is
obtained from the `TZ` environment variable or the platform timezone configuration.
* `0400@America/New_York` means 04:00 in the named IANA timezone, including daylight-saving rules.

A zoneless value such as `0400` continues to mean UTC for compatibility with existing deployments,
but is considered legacy. Prefer an explicit `Z`, offset, `@local`, or IANA timezone.

You can set the frequency start via:

* Environment variable: `DB_DUMP_FREQUENCY=60`
Expand All @@ -103,3 +114,13 @@ dump:
schedule:
begin: "+120"
```

For example, to begin at the next 04:00 in New York:

```bash
mysql-backup dump --frequency=1440 --begin=0400@America/New_York
```

`begin` determines the first run. Later runs use the configured frequency as an elapsed number of
minutes. Consequently, a frequency of 1440 may shift by one local hour after a daylight-saving
transition; use a timezone-aware cron schedule when every run must remain at the same local time.
Loading
Loading