sql >> Database >  >> RDS >> PostgreSQL

Veelvoorkomende fouten bij het migreren van PostgreSQL-databases van on-premises naar AWS RDS

In deel één, PostgreSQL-databases migreren van op locatie naar de cloud met AWS RDS, hebben we u laten zien hoe u de migratie uitvoert. In deze blog zullen we enkele veelvoorkomende fouten onderzoeken die tijdens de migratie kunnen optreden.

BACK-UPFOUT:toestemming geweigerd voor back-upschema

/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: [archiver (db)] query failed: ERROR:  permission denied for schema source_schema
pg_dump: [archiver (db)] query was: LOCK TABLE source_schema.table1 IN ACCESS SHARE MODE

RESOLUTIE - Zorg ervoor dat source_appuser de juiste toegang heeft. D.w.z. in dit geval is source_appuser eigenaar

postgres=> \c source_database
psql (10.5, server 10.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "source_database" as user "postgres".
source_database=> \dn+
                               List of schemas
     Name      |   Owner   |   Access privileges    |      Description       
---------------+-----------+------------------------+------------------------
 public        | postgres | postgres=UC/postgres+| standard public schema
               |           | =UC/postgres          |
 source_schema | postgres |                        |
(2 rows)
 
source_database=> alter schema source_schema owner to source_appuser ;
ALTER SCHEMA
 
source_database=> \dt source_schema.table*
             List of relations
    Schema     |  Name  | Type  |   Owner  
---------------+--------+-------+-----------
 source_schema | table1 | table | postgres
 source_schema | table2 | table | postgres
 source_schema | table3 | table | postgres
(3 rows)
 
source_database=> alter table source_schema.table1 owner to source_appuser ;
ALTER TABLE
source_database=> alter table source_schema.table2 owner to source_appuser ;
ALTER TABLE
source_database=> alter table source_schema.table3 owner to source_appuser ;
ALTER TABLE
source_database=> \dt source_schema.table*
                List of relations
    Schema     |  Name  | Type  |     Owner     
---------------+--------+-------+----------------
 source_schema | table1 | table | source_appuser
 source_schema | table2 | table | source_appuser
 source_schema | table3 | table | source_appuser
(3 rows)

BACK-UPFOUT:Serverversie komt niet overeen

# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: server version: 10.6; pg_dump version: 9.6.9
pg_dump: aborting because of server version mismatch

RESOLUTIE - Gebruik de juiste PostgreSQL-binary (d.w.z. /usr/pgsql-10/bin in plaats van /usr/pgsql-9.6/bin)

# Backup database...
/usr/pgsql-10/bin/pg_dump -v source_database -h onpremdbserver.domain.com -p 5432 -U source_appuser -Fd -j 100 -f /dbbackup/10/source_database   --no-owner --no-privileges
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined access methods
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
pg_dump: reading user-defined text search configurations
pg_dump: reading user-defined foreign-data wrappers
pg_dump: reading user-defined foreign servers
pg_dump: reading default privileges
pg_dump: reading user-defined collations
pg_dump: reading user-defined conversions
pg_dump: reading type casts
pg_dump: reading transforms
pg_dump: reading table inheritance information
pg_dump: reading event triggers
pg_dump: finding extension tables
pg_dump: finding inheritance relationships
pg_dump: reading column info for interesting tables
pg_dump: finding the columns and types of table "source_schema.table1"
pg_dump: finding the columns and types of table "source_schema.table2"
pg_dump: finding the columns and types of table "source_schema.table3"
pg_dump: finding default expressions of table "source_schema.table3"
pg_dump: flagging inherited columns in subtables
pg_dump: reading indexes
pg_dump: reading extended statistics
pg_dump: reading constraints
pg_dump: reading triggers
pg_dump: reading rewrite rules
pg_dump: reading policies
pg_dump: reading row security enabled for table "source_schema.table1"
pg_dump: reading policies for table "source_schema.table1"
pg_dump: reading row security enabled for table "source_schema.table2"
pg_dump: reading policies for table "source_schema.table2"
pg_dump: reading row security enabled for table "source_schema.table3_id_seq"
pg_dump: reading policies for table "source_schema.table3_id_seq"
pg_dump: reading row security enabled for table "source_schema.table3"
pg_dump: reading policies for table "source_schema.table3"
pg_dump: reading publications
pg_dump: reading publication membership
pg_dump: reading publication membership for table "source_schema.table1"
pg_dump: reading publication membership for table "source_schema.table2"
pg_dump: reading publication membership for table "source_schema.table3"
pg_dump: reading subscriptions
pg_dump: reading large objects
pg_dump: reading dependency data
pg_dump: saving encoding = UTF8
pg_dump: saving standard_conforming_strings = on
pg_dump: saving search_path =
pg_dump: saving database definition
pg_dump: dumping contents of table "source_schema.table1"
pg_dump: finished item 3797 TABLE DATA table1
pg_dump: dumping contents of table "source_schema.table3"
pg_dump: finished item 3800 TABLE DATA table3
pg_dump: dumping contents of table "source_schema.table2"
pg_dump: finished item 3798 TABLE DATA table2

HERSTELFOUT:moet de eigenaar zijn van extensie plpgsql

# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 
ERROR:  permission denied to create database

RESOLUTIE:RDS staat geen echte supergebruiker toe, maar de fout kan worden genegeerd.

HERSTELFOUT:toestemming geweigerd om database te verwijderen

ERROR:
postgres=> drop database dest_database_newdb ;
ERROR:  must be owner of database dest_database_newdb

RESOLUTIE

postgres=> grant dest_rds_superuser to postgres ;
GRANT ROLE
postgres=> drop database dest_database_newdb ;
DROP DATABASE

HERSTELFOUT:Toestemming geweigerd om rol te wijzigen

# Grant dest_rds_superuser createdb...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
alter role dest_rds_superuser createdb
ERROR:  permission denied

RESOLUTIE

postgres=> grant rds_superuser to dest_rds_superuser ;
GRANT ROLE
 
 # Grant dest_rds_superuser createdb...
 /usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"alter role dest_rds_superuser createdb"
 alter role dest_rds_superuser createdb
 ALTER ROLE

HERSTELFOUT:toestemming geweigerd om database te maken

ERROR:  permission denied to create database

RESOLUTIE

RESOLUTION: Roles must have createdb privilege otherwise to avoid to the following error:
postgres=> alter role dest_rds_superuser createdb ;

Download de whitepaper vandaag PostgreSQL-beheer en -automatisering met ClusterControlLees wat u moet weten om PostgreSQL te implementeren, bewaken, beheren en schalenDownload de whitepaper

HERSTELFOUT:database bestaat al

# Create database...
/usr/pgsql-10/bin/psql -E -e postgres -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -Udest_rds_superuser -c"create database dest_database_newdb"
create database dest_database_newdb
ERROR:  database "dest_database_newdb" already exists
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 6; 2615 20233 SCHEMA source_schema source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  schema "source_schema" already exists
    Command was: CREATE SCHEMA source_schema;
       pg_restore: processing item 1 EXTENSION plpgsql
       pg_restore: creating EXTENSION "plpgsql"
       pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
       pg_restore: creating COMMENT "EXTENSION plpgsql"
       pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
       pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
           Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
       pg_restore: processing item 197 TABLE table1
       pg_restore: creating TABLE "source_schema.table1"
       pg_restore: [archiver (db)] Error from TOC entry 197; 1259 20234 TABLE table1 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table1" already exists
           Command was: CREATE TABLE source_schema.table1 (
           id integer
       );
       pg_restore: processing item 198 TABLE table2
       pg_restore: creating TABLE "source_schema.table2"
       pg_restore: [archiver (db)] Error from TOC entry 198; 1259 20237 TABLE table2 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table2" already exists
           Command was: CREATE TABLE source_schema.table2 (
           id integer
       );
       pg_restore: processing item 200 TABLE table3
       pg_restore: creating TABLE "source_schema.table3"
       pg_restore: [archiver (db)] Error from TOC entry 200; 1259 20242 TABLE table3 source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3" already exists
           Command was: CREATE TABLE source_schema.table3 (
           id integer NOT NULL,
           name character varying
       );
       pg_restore: processing item 199 SEQUENCE table3_id_seq
       pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
       pg_restore: [archiver (db)] Error from TOC entry 199; 1259 20240 SEQUENCE table3_id_seq source_appuser
       pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3_id_seq" already exists
           Command was: CREATE SEQUENCE source_schema.table3_id_seq
           AS integer
           START WITH 1
           INCREMENT BY 1
           NO MINVALUE
           NO MAXVALUE
           CACHE 1;
       pg_restore: processing item 3809 SEQUENCE OWNED BY table3_id_seq
       pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
       pg_restore: processing item 3675 DEFAULT table3 id
       pg_restore: creating DEFAULT "source_schema.table3 id"
       pg_restore: entering main parallel loop
       pg_restore: launching item 3797 TABLE DATA table1
       pg_restore: launching item 3798 TABLE DATA table2
       pg_restore: launching item 3800 TABLE DATA table3
       pg_restore: launching item 3810 SEQUENCE SET table3_id_seq
       pg_restore: pg_restore: executing SEQUENCE SET table3_id_seq
       processing data for table "source_schema.table2"
       pg_restore: finished item 3798 TABLE DATA table2
       pg_restore: finished item 3810 SEQUENCE SET table3_id_seq
       pg_restore: processing data for table "source_schema.table3"
       pg_restore: processing data for table "source_schema.table1"
       pg_restore: finished item 3797 TABLE DATA table1
       pg_restore: finished item 3800 TABLE DATA table3
       pg_restore: finished main parallel loop
       WARNING: errors ignored on restore: 6

RESOLUTIE - Zorg ervoor dat u de bestaande database handmatig verwijdert voordat u het script uitvoert.

# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database   --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 3806 DATABASE source_database
pg_restore: processing item 3 SCHEMA public
pg_restore: creating SCHEMA "public"
pg_restore: processing item 3807 COMMENT SCHEMA public
pg_restore: creating COMMENT "SCHEMA public"
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: creating SCHEMA "source_schema"
pg_restore: processing item 1 EXTENSION plpgsql
pg_restore: creating EXTENSION "plpgsql"
pg_restore: processing item 3808 COMMENT EXTENSION plpgsql
pg_restore: creating COMMENT "EXTENSION plpgsql"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3808; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
 
 
 
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3809 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3

HERSTELFOUT:Schema bestaat al

# Create schema...
/usr/pgsql-10/bin/psql -E -e -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 dest_database_newdb -Udest_rds_superuser -c"create schema source_schema"
create schema source_schema
ERROR:  schema "source_schema" already exists
 
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 197; 1259 20234 TABLE table1 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table1" already exists
    Command was: CREATE TABLE source_schema.table1 (
    id integer
);
 
 
 
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: [archiver (db)] Error from TOC entry 198; 1259 20237 TABLE table2 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table2" already exists
    Command was: CREATE TABLE source_schema.table2 (
    id integer
);
 
  
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: [archiver (db)] Error from TOC entry 200; 1259 20242 TABLE table3 source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3" already exists
    Command was: CREATE TABLE source_schema.table3 (
    id integer NOT NULL,
    name character varying
);
 
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: [archiver (db)] Error from TOC entry 199; 1259 20240 SEQUENCE table3_id_seq source_appuser
pg_restore: [archiver (db)] could not execute query: ERROR:  relation "table3_id_seq" already exists
    Command was: CREATE SEQUENCE source_schema.table3_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;
 
 
pg_restore: processing item 3806 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3807 SEQUENCE SET table3_id_seq
pg_restore: pg_restore: processing data for table "source_schema.table2"
processing data for table "source_schema.table1"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table3"
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished main parallel loop
WARNING: errors ignored on restore: 4

RESOLUTIE - Verwijder of hernoem bestaand schema. Ik hernoem graag totdat alles is gevalideerd:

postgres=> \c dest_database_newdb
psql (10.5, server 10.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "dest_database_newdb" as user "postgres".
dest_database_newdb=> \dn
          List of schemas
     Name      |       Owner        
---------------+--------------------
 public        | postgres
 source_schema | dest_rds_superuser
(2 rows)
 
dest_database_newdb=> alter schema source_schema rename to source_schema_old;
ALTER SCHEMA
dest_database_newdb=> \dn
            List of schemas
       Name        |       Owner        
-------------------+--------------------
 public            | postgres
 source_schema_old | dest_rds_superuser
(2 rows)
 
 
# Restore database...
/usr/pgsql-10/bin/pg_restore -v -d dest_database_newdb -hdest_dbinstance.cluster-awsrdsguid.us-east-1.rds.amazonaws.com -p5432 -U dest_rds_superuser -j 120 -Fd /dbbackup/10/source_database --schema=source_schema  --no-owner --no-privileges
pg_restore: connecting to database for restore
pg_restore: processing item 3803 ENCODING ENCODING
pg_restore: processing item 3804 STDSTRINGS STDSTRINGS
pg_restore: processing item 3805 SEARCHPATH SEARCHPATH
pg_restore: processing item 6 SCHEMA source_schema
pg_restore: processing item 197 TABLE table1
pg_restore: creating TABLE "source_schema.table1"
pg_restore: processing item 198 TABLE table2
pg_restore: creating TABLE "source_schema.table2"
pg_restore: processing item 200 TABLE table3
pg_restore: creating TABLE "source_schema.table3"
pg_restore: processing item 199 SEQUENCE table3_id_seq
pg_restore: creating SEQUENCE "source_schema.table3_id_seq"
pg_restore: processing item 3806 SEQUENCE OWNED BY table3_id_seq
pg_restore: creating SEQUENCE OWNED BY "source_schema.table3_id_seq"
pg_restore: processing item 3675 DEFAULT table3 id
pg_restore: creating DEFAULT "source_schema.table3 id"
pg_restore: entering main parallel loop
pg_restore: launching item 3797 TABLE DATA table1
pg_restore: launching item 3798 TABLE DATA table2
pg_restore: launching item 3800 TABLE DATA table3
pg_restore: launching item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table1"
pg_restore: processing data for table "source_schema.table2"
pg_restore: executing SEQUENCE SET table3_id_seq
pg_restore: finished item 3807 SEQUENCE SET table3_id_seq
pg_restore: processing data for table "source_schema.table3"
pg_restore: finished item 3797 TABLE DATA table1
pg_restore: finished item 3798 TABLE DATA table2
pg_restore: finished item 3800 TABLE DATA table3
pg_restore: finished main parallel loop

  1. Voeg een procentteken toe aan een getal in MariaDB

  2. SQL-insert genereren in voor Oracle

  3. Problemen met UTF-8-tekens; wat ik zie is niet wat ik heb opgeslagen

  4. Basisprincipes van parallel programmeren met het Fork/Join Framework in Java